SAN: Stochastic Average Newton Algorithm for Minimizing Finite Sums - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

SAN: Stochastic Average Newton Algorithm for Minimizing Finite Sums

Rui Yuan
  • Fonction : Auteur
Robert M. Gower
Jiabin Chen
  • Fonction : Auteur

Résumé

We present a principled approach for designing stochastic Newton methods for solving finite sum optimization problems. Our approach has two steps. First, we rewrite the stationarity conditions as a system of nonlinear equations that associates each data point to a new row. Second, we apply a Subsampled Newton Raphson method to solve this system of nonlinear equations. Using our approach, we develop a new Stochastic Average Newton (SAN) method, which is incremental by design, in that it requires only a single data point per iteration. It is also cheap to implement when solving regularized generalized linear models, with a cost per iteration of the order of the number of the parameters. We show through numerical experiments that SAN requires no knowledge about the problem, neither parameter tuning, while remaining competitive as compared to classical variance reduced gradient methods (e.g. SAG and SVRG), incremental Newton and quasi-Newton methods (e.g. SNM, IQN).
Fichier principal
Vignette du fichier
chen22a.pdf (987.61 Ko) Télécharger le fichier
Origine : Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-03886300 , version 1 (06-12-2022)

Identifiants

Citer

Rui Yuan, Robert M. Gower, Jiabin Chen, Guillaume Garrigos. SAN: Stochastic Average Newton Algorithm for Minimizing Finite Sums. 25th International Conference on Artificial Intelligence and Statistics, 2022, Valencia, Spain. ⟨hal-03886300⟩
32 Consultations
25 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More