Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics - Laboratoire Interdisciplinaire des Sciences du Numérique
Conference Papers Year : 2024

Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics

Abstract

Automatic metrics are used as proxies to evaluate abstractive summarization systems when human annotations are too expensive. To be useful, these metrics should be fine-grained, show a high correlation with human annotations, and ideally be independent of reference quality; however, most standard evaluation metrics for summarization are reference-based, and existing reference-free metrics correlate poorly with relevance, especially on summaries of longer documents. In this paper, we introduce a reference-free metric that correlates well with human evaluated relevance, while being very cheap to compute. We show that this metric can also be used alongside reference-based metrics to improve their robustness in low quality reference settings.
Fichier principal
Vignette du fichier
acl_latex.pdf (996.95 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-04720645 , version 1 (08-10-2024)

Identifiers

Cite

Théo Gigant, Camille Guinaudeau, Marc Decombas, Frédéric Dufaux. Mitigating the Impact of Reference Quality on Evaluation of Summarization Systems with Reference-Free Metrics. The 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), Nov 2024, Miami (FL), United States. ⟨hal-04720645⟩
79 View
39 Download

Altmetric

Share

More