Learning Sparse Filters In Deep Convolutional Neural Networks With A l 1 /l 2 Pseudo-Norm - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

Learning Sparse Filters In Deep Convolutional Neural Networks With A l 1 /l 2 Pseudo-Norm

Yongzhe Yan
Thierry Chateau
Christophe Blanc

Résumé

While deep neural networks (DNNs) have proven to be efficient for numerous tasks, they come at a high memory and computation cost, thus making them impractical on resource-limited devices. However, these networks are known to contain a large number of parameters. Recent research has shown that their structure can be more compact without compromising their performance. In this paper, we present a sparsity-inducing regularization term based on the ratio l1/l2 pseudo-norm defined on the filter coefficients. By defining this pseudo-norm appropriately for the different filter kernels, and removing irrelevant filters, the number of kernels in each layer can be drastically reduced leading to very compact Deep Convolutional Neural Networks (DCNN) structures. Unlike numerous existing methods, our approach does not require an iterative retraining process and, using this regularization term, directly produces a sparse model during the training process. Furthermore, our approach is also much easier and simpler to implement than existing methods. Experimental results on MNIST and CIFAR-10 show that our approach significantly reduces the number of filters of classical models such as LeNet and VGG while reaching the same or even better accuracy than the baseline models. Moreover, the trade-off between the sparsity and the accuracy is compared to other loss regularization terms based on the l1 or l2 norm as well as the SSL [1], NISP [2] and GAL [3] methods and shows that our approach is outperforming them.
Fichier principal
Vignette du fichier
Anthony_ICPR.pdf (499.68 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03048750 , version 1 (09-12-2020)

Identifiants

Citer

Anthony Berthelier, Yongzhe Yan, Thierry Chateau, Christophe Blanc, Stefan Duffner, et al.. Learning Sparse Filters In Deep Convolutional Neural Networks With A l 1 /l 2 Pseudo-Norm. CADL 2020 : Workshop on Computational Aspects of Deep Learning - ICPR 2020, Jan 2021, Milan, Italy. ⟨10.1007/978-3-030-68763-2_50⟩. ⟨hal-03048750⟩
87 Consultations
336 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More