FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation

Résumé

Graph-to-text (G2T) generation takes a graph as input and aims to generate a fluent and faithful textual representation of the information in the graph. The task has many applications, such as dialogue generation and question answering. In this work, we investigate to what extent the G2T generation problem is solved for previously studied datasets, and how proposed metrics perform when comparing generated texts. To help address their limitations, we propose a new metric that correctly identifies factual faithfulness, i.e., given a triple (subject, predicate, object), it decides if the triple is present in a generated text. We show that our metric FactSpotter achieves the highest correlation with human annotations on data correctness, data coverage, and relevance. In addition, FactSpotter can be used as a plug-in feature to improve the factual faithfulness of existing models. Finally, we investigate if existing G2T datasets are still challenging for state-of-the-art models. Our code is available online: https: //github.com/guihuzhang/FactSpotter.
Fichier principal
Vignette du fichier
_EMNLP_2023__Evaluating_the_Factual_Faithfulness_of_Graph_to_Text_Generation_Camera.pdf (283.25 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04257838 , version 1 (25-10-2023)

Licence

Paternité

Identifiants

  • HAL Id : hal-04257838 , version 1

Citer

Kun Zhang, Oana Balalau, Ioana Manolescu. FactSpotter: Evaluating the Factual Faithfulness of Graph-to-Text Generation. Findings of EMNLP 2023 - Conference on Empirical Methods in Natural Language Processing, Dec 2023, Singapore, Singapore. ⟨hal-04257838⟩
266 Consultations
202 Téléchargements

Partager

Gmail Facebook X LinkedIn More