Towards robust complexity indices in linguistic typology - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Studies in Language Année : 2023

Towards robust complexity indices in linguistic typology

Yoon Mi Oh
François Pellegrino

Résumé

Abstract There is high hope that corpus-based approaches to language complexity will contribute to explaining linguistic diversity. Several complexity indices have consequently been proposed to compare different aspects among languages, especially in phonology and morphology. However, their robustness against changes in corpus size and content hasn’t been systematically assessed, thus impeding comparability between studies. Here, we systematically test the robustness of four complexity indices estimated from raw texts and either routinely utilized in crosslinguistic studies (Type-Token Ratio and word-level Entropy) or more recently proposed (Word Information Density and Lexical Diversity). Our results on 47 languages strongly suggest that traditional indices are more prone to fluctuation than the newer ones. Additionally, we confirm with Word Information Density the existence of a cross-linguistic trade-off between word-internal and across-word distributions of information. Finally, we implement a proof of concept suggesting that modern deep-learning language models can improve the comparability across languages with non-parallel datasets.
Fichier principal
Vignette du fichier
sl.22034.oh.pdf (3.11 Mo) Télécharger le fichier
Origine : Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-03916272 , version 1 (30-12-2022)

Identifiants

Citer

Yoon Mi Oh, François Pellegrino. Towards robust complexity indices in linguistic typology: A corpus-based assessment. Studies in Language, 2023, 47 (4), pp.789-829. ⟨10.1075/sl.22034.oh⟩. ⟨hal-03916272⟩
18 Consultations
24 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More