Pré-Publication, Document De Travail Année : 2024

A second-order-like optimizer with adaptive gradient scaling for deep learning

Résumé

In this empirical article, we introduce INNAprop, an optimization algorithm that combines the INNA method with the RMSprop adaptive gradient scaling. It leverages second-order information and rescaling while keeping the memory requirements of standard DL methods as AdamW or SGD with momentum. After having recalled our geometrical motivations, we provide quite extensive experiments. On image classification (CIFAR-10, ImageNet) and language modeling (GPT-2), INNAprop consistently matches or outperforms AdamW both in training speed and accuracy, with minimal hyperparameter tuning in large-scale settings. Our code is publicly available at \url{https://github.com/innaprop/innaprop}.

Fichier principal
Vignette du fichier
innaprop/innaprop_arxiv.pdf (971.87 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04724894 , version 1 (07-10-2024)
hal-04724894 , version 2 (09-12-2024)
hal-04724894 , version 3 (10-12-2024)

Licence

Identifiants

Citer

Jérôme Bolte, Ryan Boustany, Edouard Pauwels, Andrei Purica. A second-order-like optimizer with adaptive gradient scaling for deep learning. 2024. ⟨hal-04724894v1⟩
699 Consultations
421 Téléchargements

Altmetric

Partager

  • More