Communication Dans Un Congrès Année : 2025

On the Representations of Entities in Auto-regressive Large Language Models

Résumé

Named entities are fundamental building blocks of knowledge in text, grounding factual information and structuring relationships within language. Despite their importance, it remains unclear how Large Language Models (LLMs) internally represent entities. Prior research has primarily examined explicit relationships, but little is known about entity representations themselves. We introduce entity mention reconstruction as a novel framework for studying how LLMs encode and manipulate entities. We investigate whether entity mentions can be generated from internal representations, how multi-token entities are encoded beyond last-token embeddings, and whether these representations capture relational knowledge. Our proposed method, leveraging task vectors, allows to consistently generate multi-token mentions from various entity representations derived from the LLMs hidden states. We thus introduce the Entity Lens, extending the logit-lens to predict multi-token mentions. Our results bring new evidence that LLMs develop entity-specific mechanisms to represent and manipulate any multi-token entities, including those unseen during training.

Fichier principal
Vignette du fichier
2510.09421v1.pdf (1.23 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05375570 , version 1 (21-11-2025)

Licence

Identifiants

Citer

Victor Morand, Josiane Mothe, Benjamin Piwowarski. On the Representations of Entities in Auto-regressive Large Language Models. 8th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, Nov 2025, Suzhou, China. pp.433-451, ⟨10.18653/v1/2025.blackboxnlp-1.25⟩. ⟨hal-05375570⟩
2228 Consultations
323 Téléchargements

Altmetric

Partager

  • More