Module-wise Training of Neural Networks via the Minimizing Movement Scheme - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Module-wise Training of Neural Networks via the Minimizing Movement Scheme

Résumé

Greedy layer-wise or module-wise training of neural networks is compelling in constrained and on-device settings where memory is limited, as it circumvents a number of problems of end-to-end back-propagation. However, it suffers from a stagnation problem, whereby early layers overfit and deeper layers stop increasing the test accuracy after a certain depth. We propose to solve this issue by introducing a module-wise regularization inspired by the minimizing movement scheme for gradient flows in distribution space. We call the method TRGL for Transport Regularized Greedy Learning and study it theoretically, proving that it leads to greedy modules that are regular and that progressively solve the task. Experimentally, we show improved accuracy of module-wise training of various architectures such as ResNets, Transformers and VGG, when our regularization is added, superior to that of other module-wise training methods and often to end-to-end training, with as much as 60% less memory usage.
Fichier principal
Vignette du fichier
gqrjrpvtwwmfbdgrpmpkbnprrxvtwcyz.pdf (1014.27 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04223364 , version 1 (29-09-2023)

Identifiants

  • HAL Id : hal-04223364 , version 1

Citer

Skander Karkar, Ibrahim Ayed, Emmanuel de Bézenac, Patrick Gallinari. Module-wise Training of Neural Networks via the Minimizing Movement Scheme. Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS 2023), Dec 2023, New Orleans (Louisiana), United States. ⟨hal-04223364⟩
46 Consultations
83 Téléchargements

Partager

Gmail Facebook X LinkedIn More