Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2020

Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification

Résumé

We analyze in a closed form the learning dynamics of stochastic gradient descent (SGD) for a single layer neural network classifying a high-dimensional Gaussian mixture where each cluster is assigned one of two labels. This problem provides a prototype of a non-convex loss landscape with interpolating regimes and a large generalization gap. We define a particular stochastic process for which SGD can be extended to a continuous-time limit that we call stochastic gradient flow. In the full-batch limit we recover the standard gradient flow. We apply dynamical mean-field theory from statistical physics to track the dynamics of the algorithm in the high-dimensional limit via a self-consistent stochastic process. We explore the performance of the algorithm as a function of control parameters shedding light on how it navigates the loss landscape.
Fichier principal
Vignette du fichier
hal_submission_neurips.pdf (1.56 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-02983444 , version 1 (22-11-2021)

Identifiants

  • HAL Id : hal-02983444 , version 1

Citer

Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, Lenka Zdeborová. Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification. Advances in Neural Information Processing Systems, Dec 2020, Online, Canada. ⟨hal-02983444⟩
79 Consultations
43 Téléchargements

Partager

Gmail Facebook X LinkedIn More