Syntax and Geometry of Information - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Syntax and Geometry of Information

La syntaxe et la géométrie de l'information

Résumé

This paper presents an information-theoretical model of syntactic generalization. We study syntactic generalization from the perspective of the capacity to disentangle semantic and structural information, emulating the human capacity to assign a grammaticality judgment to semantically nonsensical sentences. In order to isolate the structure, we propose to represent the probability distribution behind a corpus as the product of the probability of a semantic context and the probability of a structure, the latter being independent of the former. We further elaborate the notion of abstraction as a relaxation of the property of independence. It is based on the measure of structural and contextual information for a given representation. We test abstraction as an optimization objective on the task of inducing syntactic categories from natural language data and show that it significantly outperforms alternative methods. Furthermore, we find that when syntax-unaware optimization objectives succeed in the task, their success is mainly due to an implicit disentanglement process rather than to the model structure. On the other hand, syntactic categories can be deduced in a principled way from the independence between structure and context.
Fichier principal
Vignette du fichier
ACL23_finalKata-2.pdf (418.66 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04125067 , version 1 (11-06-2023)

Licence

Paternité

Identifiants

  • HAL Id : hal-04125067 , version 1

Citer

Raphaël Bailly, Laurent Leblond, Kata Gábor. Syntax and Geometry of Information. ACL '23: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Jul 2023, Toronto, Canada. ⟨hal-04125067⟩
33 Consultations
35 Téléchargements

Partager

Gmail Facebook X LinkedIn More