Anisotropy Is Inherent to Self-Attention in Transformers - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2024

Anisotropy Is Inherent to Self-Attention in Transformers

Résumé

The representation degeneration problem is a phenomenon that is widely observed among self-supervised learning methods based on Transformers. In NLP, it takes the form of anisotropy, a singular property of hidden representations which makes them unexpectedly close to each other in terms of angular distance (cosine-similarity). Some recent works tend to show that anisotropy is a consequence of optimizing the cross-entropy loss on long-tailed distributions of tokens. We show in this paper that anisotropy can also be observed empirically in language models with specific objectives that should not suffer directly from the same consequences. We also show that the anisotropy problem extends to Transformers trained on other modalities. Our observations tend to demonstrate that anisotropy might actually be inherent to Transformers-based models.
Fichier principal
Vignette du fichier
eacl_2024.pdf (6.28 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04593391 , version 1 (29-05-2024)

Identifiants

Citer

Nathan Godey, Eric Villemonte de La Clergerie, Benoît Sagot. Anisotropy Is Inherent to Self-Attention in Transformers. EACL 2024 - 18th Conference of the European Chapter of the Association for Computational Linguistics, Mar 2024, St Julians, Malta. pp.35-48. ⟨hal-04593391⟩
8 Consultations
7 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More