Communication Dans Un Congrès Année : 2025

Transformative or Conservative? Conservation laws for ResNets and Transformers

Résumé

While conservation laws in gradient flow training dynamics are well understood for (mostly shallow) ReLU and linear networks, their study remains largely unexplored for more practical architectures. This paper bridges this gap by deriving and analyzing conservation laws for modern architectures, with a focus on convolutional ResNets and Transformer networks. For this, we first show that basic building blocks such as ReLU (or linear) shallow networks, with or without convolution, have easily expressed conservation laws, and no more than the known ones. In the case of a single attention layer, we also completely describe all conservation laws, and we show that residual blocks have the same conservation laws as the same block without skip connection. We then introduce the notion of conservation laws that depend only on a subset of parameters (corresponding e.g. to a pair of consecutive layers, to a residual block, or to an attention layer). We demonstrate that the characterization of such laws can be reduced to the analysis of the corresponding building block in isolation. Finally, we examine how these newly discovered conservation principles, initially established in the continuous gradient flow regime, persist under discrete optimization dynamics, particularly in the context of Stochastic Gradient Descent (SGD).

Fichier principal
Vignette du fichier
ICML25 (8).pdf (2.26 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05108418 , version 1 (11-06-2025)

Licence

Identifiants

  • HAL Id : hal-05108418 , version 1

Citer

Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré. Transformative or Conservative? Conservation laws for ResNets and Transformers. 42nd International Conference on Machine Learning (ICML 2025), Jul 2025, Vancouver (Canada), Canada. ⟨hal-05108418⟩
1732 Consultations
344 Téléchargements

Partager

  • More