Communication Dans Un Congrès Année : 2024

Evaluating the Generalisation of an Artificial Learner

Evaluation de la généralisabilité d'un apprenant artificiel.

Résumé

This paper focused on the creation of LLM-based artificial learners. Motivated by the capability of language models to encode language representation, we evaluated such models for predicting masked tokens in learner corpora. We domain-adapted the BERT model, pre-trained on native English, by further pre- training two learner models on learner corpora: a natural learner model on the EFCAM- DAT dataset and a synthetic learner model on the C4200m dataset. We evaluated the two artificial learner models alongside the baseline native model using an external English-for-specific-purposes corpus from French undergraduates. We evaluated metrics related to accuracy, consistency, and divergence. While the native model performed reasonably well, the natural learner pre-trained model showed improvements in recall-at-k. We analysed error patterns, showing that the native model made “overconfident” errors by assigning high probabilities to incorrect predictions, while the artificial learners distributed probabilities more evenly when wrong. Finally, we showed that the general token choices from the native model diverged from the natural learner model and this divergence was higher at lower proficiency levels.

Cet article évalue la possibilité d'utiliser des LLM pour simuler le comportement des apprenants francophones de l'anglais à partir d'une version de BERT ré-entraînée avec des données du corpus EFCAMDAT. Nous comparons les performances de ce modèle avec une version de BERT.

Fichier principal
Vignette du fichier
2024.nlp4call-1.15-1.pdf (452.55 Ko) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte
Licence

Dates et versions

hal-04862076 , version 1 (02-01-2025)

Licence

Identifiants

  • HAL Id : hal-04862076 , version 1

Citer

Bernardo Stearns, Nicolas Ballier, Thomas Gaillat, Andrew J. Simpkin, John P. Mc Crae. Evaluating the Generalisation of an Artificial Learner. NLP4CALL2024 : Natural Language Processing for Computer-assisted Language Learning, Université Rennes 2, France; University of Gothenburg, Sweden; Linköping University, Sweden, Oct 2024, Rennes, France. ⟨hal-04862076⟩
138 Consultations
126 Téléchargements

Partager

  • More