A Methodology for the Comparison of Human Judgments With Metrics for Coreference Resolution - Archive ouverte HAL Access content directly
Conference Papers Year : 2022

A Methodology for the Comparison of Human Judgments With Metrics for Coreference Resolution

Abstract

We propose a method for investigating the interpretability of metrics used for the coreference resolution task through comparisons with human judgments. We provide a corpus with annotations of different error types and human evaluations of their gravity. Our preliminary analysis shows that metrics considerably overlook several error types and overlook errors in general in comparison to humans. This study is conducted on French texts, but the methodology should be language-independent.
Fichier principal
Vignette du fichier
A_Methodology_for_the_Comparison_of_Human_Judgments_With_Metrics_for_Coreference_Resolution.pdf (176.27 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03650294 , version 1 (24-04-2022)

Identifiers

Cite

Mariya Borovikova, Loïc Grobol, Anaïs Halftermeyer, Sylvie Billot. A Methodology for the Comparison of Human Judgments With Metrics for Coreference Resolution. HumEval at ACL, May 2022, Dublin, Ireland. pp.16-23, ⟨10.18653/v1/2022.humeval-1.2⟩. ⟨hal-03650294⟩
111 View
96 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More