Path-metrics, pruning, and generalization - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2024

Path-metrics, pruning, and generalization

Résumé

Analyzing the behavior of ReLU neural networks often hinges on understanding the relationships between their parameters and the functions they implement. This paper proves a new bound on function distances in terms of the so-called path-metrics of the parameters. Since this bound is intrinsically invariant with respect to the rescaling symmetries of the networks, it sharpens previously known bounds. It is also, to the best of our knowledge, the first bound of its kind that is broadly applicable to modern networks such as ResNets, VGGs, U-nets, and many more. In contexts such as network pruning and quantization, the proposed path-metrics can be efficiently computed using only two forward passes. Besides its intrinsic theoretical interest, the bound yields not only novel theoretical generalization bounds, but also a promising proof of concept for rescaling-invariant pruning.

Domaines

Autres [stat.ML]
Fichier principal
Vignette du fichier
neurips_2024.pdf (1.24 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04584311 , version 1 (23-05-2024)
hal-04584311 , version 2 (23-05-2024)

Identifiants

  • HAL Id : hal-04584311 , version 1

Citer

Antoine Gonon, Nicolas Brisebarre, Elisa Riccietti, Rémi Gribonval. Path-metrics, pruning, and generalization. 2024. ⟨hal-04584311v1⟩
0 Consultations
0 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More