Convolutional neural networks contain structured strong lottery tickets - Archive ouverte HAL
Communication Dans Un Congrès Année : 2023

Convolutional neural networks contain structured strong lottery tickets

Résumé

The Strong Lottery Ticket Hypothesis (SLTH) states that randomly-initialised neural networks contain subnetworks that can perform well without any training. Although unstructured pruning has been extensively studied in this context, its structured counterpart, which can deliver significant computational and memory efficiency gains, has been largely unexplored. One of the main reasons for this gap is the limitations of the underlying mathematical tools used in formal analyses of the SLTH. In this paper, we overcome these limitations: we leverage recent advances in the multidimensional generalisation of the Random Subset-Sum Problem and obtain a variant that admits the stochastic dependencies that arise when addressing structured pruning in the SLTH. We apply this result to prove, for a wide class of random Convolutional Neural Networks, the existence of structured subnetworks that can approximate any sufficiently smaller network. This is the first work to address the SLTH for structured pruning, opening up new avenues for further research on the hypothesis and contributing to the understanding of the role of overparameterization in deep learning.
Fichier principal
Vignette du fichier
kernelpruning.pdf (431.61 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04143024 , version 1 (27-06-2023)
hal-04143024 , version 2 (16-11-2023)

Identifiants

  • HAL Id : hal-04143024 , version 1

Citer

Arthur Carvalho Walraven da Cunha, Francesco d'Amore, Emanuele Natale. Convolutional neural networks contain structured strong lottery tickets. 37th Conference on Neural Information Processing Systems (NeurIPS 2023), Nov 2023, New Orleans, United States. ⟨hal-04143024v1⟩
241 Consultations
178 Téléchargements

Partager

More