Stochastic Differential Equations for modeling first order optimization methods - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2024

Stochastic Differential Equations for modeling first order optimization methods

Résumé

In this article, a family of SDEs are derived as a tool to understand the behavior of numerical optimization methods under random evaluations of the gradient. Our objective is to transpose the introduction of continuous version through ODEs to understand the asymptotic behavior of discrete optimization scheme to the stochastic setting. We consider a continuous version of the stochastic gradient scheme and of a stochastic inertial system. This article first studies the quality of the approximation of the discrete scheme by a SDE when the step size tends to 0. Then, it presents new asymptotic bounds on the values F (X(t)) − F * where X(t) is a solution of the SDE and F * = min F , when F is convex and under integrability conditions on the noise. Results are provided under two sets of hypotheses : first considering C 2 and convex functions and then adding some geometrical properties of F. All these results give an insight on the behavior of these inertial and perturbed algorithms in the setting of stochastic algorithms.
Fichier principal
Vignette du fichier
finalversionM143566.pdf (406.97 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03630785 , version 1 (05-04-2022)
hal-03630785 , version 2 (12-06-2024)

Identifiants

  • HAL Id : hal-03630785 , version 2

Citer

Marc Dambrine, Ch Dossal, Bénédicte Puig, Aude Rondepierre. Stochastic Differential Equations for modeling first order optimization methods. 2024. ⟨hal-03630785v2⟩
129 Consultations
439 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More