Communication Dans Un Congrès Année : 2025

Semantic Evaluation of Multilingual Knowledge Graph-to-Text Generation via NLI Fine-Tuning: Precision, Recall and F1 scores

Résumé

Performance in the Knowledge Graph-to-Text generation has improved over time, particularly in English. However, models are still prone to mistakes like Additions and Omissions. Furthermore, few languages are taken into account since both train and test data are not readily available. In this paper, we hope to facilitate the development and improvement of multilingual KG-to-Text models by providing a multilingual evaluation framework that is referenceless and permits estimating how much a KGto-Text Model under-(omission) or over-(addition) generates. We focus on two high (English, Russian) and five low (Breton, Irish, Maltese, Welsh, Xhosa) resource languages and show that our metric has fair to moderate correlation with reference-based metrics, positioning it as a consistent alternative when no references are available. We also show that our metric outperforms prior reference-less metrics in correlation with existing human judgments. Additional human evaluation shows moderate to strong correlation with human annotators in assessing precision and recall at a higher granularity level than shown in previous studies. Since our metric provides scores for precision and recall, it helps better assess the level of over-or undergeneration of multilingual KG-to-Text models. We make our data, code and models available.

Fichier principal
Vignette du fichier
nli_multilingual_metrics_william_acl25 (1).pdf (4.14 Mo) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte
Licence

Dates et versions

hal-05138142 , version 1 (01-07-2025)

Licence

Identifiants

  • HAL Id : hal-05138142 , version 1

Citer

William Soto Martinez, Yannick Parmentier, Claire Gardent. Semantic Evaluation of Multilingual Knowledge Graph-to-Text Generation via NLI Fine-Tuning: Precision, Recall and F1 scores. The 63rd Meeting of the Association for Computational Linguistics, ACL, Jul 2025, Vienna, Austria. ⟨hal-05138142⟩
308 Consultations
292 Téléchargements

Partager

  • More