Demystifying Softmax Gating Function in Gaussian Mixture of Experts - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Demystifying Softmax Gating Function in Gaussian Mixture of Experts

Résumé

Understanding parameter estimation of softmax gating Gaussian mixture of experts has remained a long-standing open problem in the literature. It is mainly due to three fundamental theoretical challenges associated with the softmax gating: (i) the identifiability only up to the translation of the parameters; (ii) the intrinsic interaction via partial differential equation between the softmax gating and the expert functions in Gaussian distribution; (iii) the complex dependence between the numerator and denominator of the conditional density of softmax gating Gaussian mixture of experts. We resolve these challenges by proposing novel Vononoi loss functions among parameters and establishing the convergence rates of the maximum likelihood estimator (MLE) for solving parameter estimation in these models. When the number of experts is unknown and over-specified, our findings show a connection between the rate of MLE and a solvability problem of a system of polynomial equations.

Dates et versions

hal-04125060 , version 1 (11-06-2023)

Licence

Paternité

Identifiants

Citer

Huy Nguyen, Trungtin Nguyen, Nhat Ho. Demystifying Softmax Gating Function in Gaussian Mixture of Experts. Advances in Neural Information Processing Systems, NeurIPS 2023 Spotlight, Acceptance rate 3.6% over 12343 submissions, Dec 2023, New Orleans, United States. pp.1-27. ⟨hal-04125060⟩
59 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More