PHYLOGENETIC REPLAY LEARNING IN DEEP NEURAL NETWORKS - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Jordanian Journal of Computers and Information Technology Année : 2022

PHYLOGENETIC REPLAY LEARNING IN DEEP NEURAL NETWORKS

Jean-Patrice Glafkides
Herman Akdag
  • Fonction : Auteur
  • PersonId : 1135814

Résumé

Though substantial advancements have been made in training deep neural networks, one problem remains, the vanishing gradient. The very strength of deep neural networks, their depth, is also unfortunately their problem, due to the difficulty of thoroughly training the deeper layers due to the vanishing gradient. This paper proposes "Phylogenetic Replay Learning", a learning methodology that substantially alleviates the vanishing-gradient problem. Unlike the residual learning methods, it does not restrict the structure of the model. Instead, it leverages elements from neuroevolution, transfer learning and layer-by-layer training. We demonstrate that this new approach is able to produce a better performing model and by calculating Shannon entropy of weights, we show that the deeper layers are trained much more thoroughly and contain statistically significantly more information than when a model is trained in a traditional brute force manner.
Fichier principal
Vignette du fichier
Glafkides.pdf (1.42 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04426845 , version 1 (01-02-2024)

Identifiants

  • HAL Id : hal-04426845 , version 1

Citer

Jean-Patrice Glafkides, Gene I Sher, Herman Akdag. PHYLOGENETIC REPLAY LEARNING IN DEEP NEURAL NETWORKS. Jordanian Journal of Computers and Information Technology, 2022, 8 (2), pp.112-126. ⟨hal-04426845⟩
21 Consultations
7 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More