Learning Sparse deep neural networks using efficient structured projections on convex constraints for green AI - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2020

Learning Sparse deep neural networks using efficient structured projections on convex constraints for green AI

Michel Barlaud
Frederic Guyard
  • Fonction : Auteur
  • PersonId : 1068789

Résumé

Deep neural networks (DNN) have been applied recently to different domains and perform better than classical state-of-the-art methods. However the high level of performances of DNNs is most often obtained with networks containing millions of parameters and for which training requires substantial computational power. To deal with this computational issue proximal regularization methods have been proposed in the literature but they are time consuming.\\ In this paper, we propose instead a constrained approach. We provide the general framework for this new projection gradient method. Our algorithm iterates a gradient step and a projection on convex constraints. We studied algorithms for different constraints: the classical $\ell_1$ unstructured constraint and structured constraints such as the $\ell_{2,1} $ constraint (Group LASSO). We propose a new $\ell_{1,1} $ structured constraint for which we provide a new projection algorithm. Finally, we used the recent "Lottery optimizer" replacing the threshold by our $\ell_{1,1} $ projection. We demonstrate the effectiveness of this method with three popular datasets (MNIST, Fashion MNIST and CIFAR). Experiments with these datasets show that our projection method using this new $\ell_{1,1} $ structured constraint provides the best decrease in memory and computational power.
Fichier principal
Vignette du fichier
DNN-ICPR-Milan.pdf (613.83 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02556382 , version 1 (28-04-2020)
hal-02556382 , version 2 (16-09-2020)
hal-02556382 , version 3 (28-10-2020)

Identifiants

  • HAL Id : hal-02556382 , version 3

Citer

Michel Barlaud, Frederic Guyard. Learning Sparse deep neural networks using efficient structured projections on convex constraints for green AI. ICPR 2020, 2020, Milan, Italy. ⟨hal-02556382v3⟩
270 Consultations
1711 Téléchargements

Partager

Gmail Facebook X LinkedIn More