Understanding and Training Deep Diagonal Circulant Neural Networks - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2020

Understanding and Training Deep Diagonal Circulant Neural Networks

Alexandre Araujo
  • Fonction : Auteur
  • PersonId : 1044588
Benjamin Negrevergne
Jamal Atif

Résumé

In this paper, we study deep diagonal circulant neural networks, which are deep neural networks in which weight matrices are the product of diagonal and circulant ones. Besides making a theoretical analysis of their expressivity, we introduce principled techniques for training these models: we devise an initialization scheme and propose a smart use of non-linearity functions in order to train deep diagonal circulant networks. Furthermore, we show that these networks outperform recently introduced deep networks with other types of structured layers. We conduct a thorough experimental study to compare the performance of deep diagonal circulant networks with state-of-the-art models based on structured matrices and with dense models. We show that our models achieve better accuracy than other structured approaches while requiring 2x fewer weights than the next best approach. Finally, we train compact and accurate deep diagonal circulant networks on a real world video classification dataset with over 3.8 million training examples.
Fichier principal
Vignette du fichier
628_paper.pdf (434.55 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03916848 , version 1 (31-12-2022)

Identifiants

  • HAL Id : hal-03916848 , version 1

Citer

Alexandre Araujo, Benjamin Negrevergne, Yann Chevaleyre, Jamal Atif. Understanding and Training Deep Diagonal Circulant Neural Networks. 24th European Conference on Artificial Intelligence (ECAI), Jul 2020, Santiago, Spain. ⟨hal-03916848⟩
5 Consultations
33 Téléchargements

Partager

Gmail Facebook X LinkedIn More