Path-metrics, pruning, and generalization - Archive ouverte HAL Access content directly
Preprints, Working Papers, ... Year : 2024

Path-metrics, pruning, and generalization

Abstract

Analyzing the behavior of ReLU neural networks often hinges on understanding the relationships between their parameters and the functions they implement. This paper proves a new bound on function distances in terms of the so-called path-metrics of the parameters. Since this bound is intrinsically invariant with respect to the rescaling symmetries of the networks, it sharpens previously known bounds. It is also, to the best of our knowledge, the first bound of its kind that is broadly applicable to modern networks such as ResNets, VGGs, U-nets, and many more. In contexts such as network pruning and quantization, the proposed path-metrics can be efficiently computed using only two forward passes. Besides its intrinsic theoretical interest, the bound yields not only novel theoretical generalization bounds, but also a promising proof of concept for rescaling-invariant pruning.
Fichier principal
Vignette du fichier
neurips_2024.pdf (1.22 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04584311 , version 1 (23-05-2024)
hal-04584311 , version 2 (23-05-2024)

Identifiers

  • HAL Id : hal-04584311 , version 2

Cite

Antoine Gonon, Nicolas Brisebarre, Elisa Riccietti, Rémi Gribonval. Path-metrics, pruning, and generalization. 2024. ⟨hal-04584311v2⟩
0 View
0 Download

Share

Gmail Mastodon Facebook X LinkedIn More