Nonlinear Acceleration of Momentum and Primal-Dual Algorithms - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Mathematical Programming, Series A Année : 2022

Nonlinear Acceleration of Momentum and Primal-Dual Algorithms

Résumé

We describe a convergence acceleration scheme for multistep optimization algorithms. The extrapolated solution is written as a nonlinear average of the iterates produced by the original optimization algorithm. Our scheme does not need the underlying fixed-point operator to be symmetric, hence handles e.g. algorithms with momentum terms such as Nesterov's accelerated method, or primal-dual methods. The weights are computed via a simple linear system and we analyze performance in both online and offline modes. We use Crouzeix's conjecture to show that acceleration performance is controlled by the solution of a Chebyshev problem on the numerical range of a non-symmetric operator modelling the behavior of iterates near the optimum. Numerical experiments are detailed on image processing problems, logistic regression and neural network training for CIFAR10 and ImageNet.

Dates et versions

hal-01893921 , version 1 (11-10-2018)

Identifiants

Citer

Raghu Bollapragada, Damien Scieur, Alexandre d'Aspremont. Nonlinear Acceleration of Momentum and Primal-Dual Algorithms. Mathematical Programming, Series A, 2022, pp.1-38. ⟨10.1007/s10107-022-01775-x⟩. ⟨hal-01893921⟩
82 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More