Applying maximum entropy principle on quantized neural networks correlates with high accuracy - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2024

Applying maximum entropy principle on quantized neural networks correlates with high accuracy

Cyril Moineau
  • Fonction : Directeur scientifique
  • PersonId : 1339524
Olivier Bichler
  • Fonction : Directeur scientifique
  • PersonId : 936077
Fabrice Rastello

Résumé

Neural network quantization, sparsification or Neural Architecture Search (NAS) have shown great success in reducing model size and computational cost on many different ML tasks and architectures. Appropriately reducing model size without degrading task performance requires a suitable measure for quantifying the amount of important information entailed in the model parameters. Information theory provides such a tool, called entropy, which, from a probabilistic point of view, makes it possible to determine whether a model has effectively extracted important information from the data. Based on information theory and the maximum entropy principle, this paper investigates the influence of the clamping function on the distribution of weights in a quantized neural network. We show that the entropy weights can be increased by using a trainable parameter that evolves during training. We also identify a correlation between high entropy and high performance. Finally, by using a regularizer that enables the model to further increase its entropy we highlighted the importance to split training into a generalization phase and an information optimization phase.
Fichier principal
Vignette du fichier
6th_AccML_paper_6.pdf (692.3 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04409740 , version 1 (22-01-2024)

Identifiants

  • HAL Id : hal-04409740 , version 1

Citer

Lucas Maisonnave, Cyril Moineau, Olivier Bichler, Fabrice Rastello. Applying maximum entropy principle on quantized neural networks correlates with high accuracy. AccML 2024 - 6th Workshop on Accelerated Machine Learning, Jan 2024, Munich, Germany. ⟨hal-04409740⟩
52 Consultations
111 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More