Rethinking the Variational Interpretation of Accelerated Optimization Methods - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Rethinking the Variational Interpretation of Accelerated Optimization Methods

Résumé

The continuous-time model of Nesterov’s momentum provides a thought-provoking perspective for understanding the nature of the acceleration phenomenon in convex optimization. One of the main ideas in this line of research comes from the field of classical mechanics and proposes to link Nesterov’s trajectory to the solution of a set of Euler-Lagrange equations relative to the so-called Bregman Lagrangian. In the last years, this approach led to the discovery of many new (stochastic) accelerated algorithms and provided a solid theoretical foundation for the design of structure-preserving accelerated methods. In this work, we revisit this idea and provide an in-depth analysis of the action relative to the Bregman Lagrangian from the point of view of calculus of variations. Our main finding is that while Nesterov’s method is a stationary point for the action, it is often not a minimizer but instead a saddle point for this functional in the space of differentiable curves. This finding challenges the main intuition behind the variational interpretation of Nesterov’s method and provides additional insights into the intriguing geometry of accelerated paths.
Fichier principal
Vignette du fichier
NeurIPS-2021-rethinking-the-variational-interpretation-of-accelerated-optimization-methods-Paper (1).pdf (876.39 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03454342 , version 1 (29-11-2021)

Identifiants

  • HAL Id : hal-03454342 , version 1

Citer

Peiyuan Zhang, Antonio Orvieto, Hadi Daneshmand. Rethinking the Variational Interpretation of Accelerated Optimization Methods. NeurrIPS 2021, Nov 2021, Virtual, Unknown Region. ⟨hal-03454342⟩
39 Consultations
88 Téléchargements

Partager

Gmail Facebook X LinkedIn More