Challenging the "One Single Vector per Token" Assumption - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Challenging the "One Single Vector per Token" Assumption

Résumé

In this paper we question the almost universal assumption that in neural networks each token should be represented by a single vector. In fact, it is so natural to use one vector per word that most people do not even consider it as an assumption of their various models. Via a series of experiments on dependency parsing, in which we let each token in a sentence be represented by a sequence of vectors, we show that the "one single vector per token" assumption might be too strong for recurrent neural networks. Indeed, biaffine parsers seem to work better when their encoder accesses its input's tokens' representations in several time steps rather than all at once. This seems to indicate that having only one occasion to look at a token through its vector is too strong a constraint for recurrent neural networks and calls for further studies on the way tokens are fed to neural networks.
Fichier principal
Vignette du fichier
2023.conll-1.33.pdf (289.73 Ko) Télécharger le fichier
Origine : Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-04334826 , version 1 (11-12-2023)

Identifiants

  • HAL Id : hal-04334826 , version 1

Citer

Mathieu Dehouck. Challenging the "One Single Vector per Token" Assumption. The SIGNLL Conference on Computational Natural Language Learning, Dec 2023, Singapore, Singapore. pp.498-507. ⟨hal-04334826⟩
26 Consultations
21 Téléchargements

Partager

Gmail Facebook X LinkedIn More