Communication Dans Un Congrès Année : 2025

Factual Knowledge Assessment of Language Models Using Distractors

Résumé

Language models encode extensive factual knowledge within their parameters. The accurate assessment of this knowledge is crucial for understanding and improving these models. In the literature, factual knowledge assessment often relies on cloze sentences, which can lead to erroneous conclusions due to the complexity of natural language (out-of-subject continuations, the existence of many correct answers and the several ways of expressing them). In this paper, we introduce a new interpretable knowledge assessment method that mitigates these issues by leveraging distractors-incorrect but plausible alternatives to the correct answer. We propose several strategies for retrieving distractors and determine the most effective one through experimentation. Our method is evaluated against existing approaches, demonstrating solid alignment with human judgment and stronger robustness to verbalization artifacts. The code and data to reproduce our experiments are available on GitHub * .

Fichier principal
Vignette du fichier
Factual_Knowledge_Assessment_For_Language_Models_using_Distractors (1).pdf (441.14 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04924313 , version 1 (17-03-2025)

Licence

Identifiants

  • HAL Id : hal-04924313 , version 1

Citer

Hichem Ammar Khodja, Abderrahmane Ait Gueni Ssaid, Frédéric Béchet, Quentin Brabant, Alexis Nasr, et al.. Factual Knowledge Assessment of Language Models Using Distractors. COLING 2025, Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, Steven Schockaert, Jan 2025, Abou Dhabi, United Arab Emirates. pp.8043-8056. ⟨hal-04924313⟩
106 Consultations
152 Téléchargements

Partager

  • More