Pré-Publication, Document De Travail Année : 2025

Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility

Résumé

The achievements of Large Language Models in Natural Language Processing, especially for high-resource languages, call for a better understanding of their characteristics from a cognitive perspective. Researchers have attempted to evaluate artificial models by testing their ability to predict behavioral (e.g., eye-tracking fixations) and physiological (e.g., brain responses) variables during language processing (e.g., reading/listening). In this paper, we propose using spontaneous speech corpora to derive production variables (speech reductions, prosodic prominences) and applying them in a similar fashion. More precisely, we extract. We then test models trained with a standard procedure on different pretraining datasets (written, spoken, and mixed genres) for their ability to predict these two variables. Our results show that, after some fine-tuning, the models can predict these production variables well above baselines. We also observe that spoken genre training data provides more accurate predictions than written genres. These results contribute to the broader effort of using high-quality speech corpora as benchmarks for LLMs.

Fichier principal
Vignette du fichier
2505.16277v1.pdf (870.08 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05266194 , version 1 (17-02-2026)

Licence

Identifiants

Citer

Sheng-Fu Wang, Laurent Prevot, Jou-An Chi, Ri-Sheng Huang, Shu-Kai Hsieh. Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility. 2026. ⟨hal-05266194⟩
66 Consultations
54 Téléchargements

Altmetric

Partager

  • More