A Methodology for the Comparison of Human Judgments With Metrics for Coreference Resolution - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

A Methodology for the Comparison of Human Judgments With Metrics for Coreference Resolution

Résumé

We propose a method for investigating the interpretability of metrics used for the coreference resolution task through comparisons with human judgments. We provide a corpus with annotations of different error types and human evaluations of their gravity. Our preliminary analysis shows that metrics considerably overlook several error types and overlook errors in general in comparison to humans. This study is conducted on French texts, but the methodology should be language-independent.
Fichier principal
Vignette du fichier
A_Methodology_for_the_Comparison_of_Human_Judgments_With_Metrics_for_Coreference_Resolution.pdf (176.27 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03650294 , version 1 (24-04-2022)

Identifiants

Citer

Mariya Borovikova, Loïc Grobol, Anaïs Halftermeyer, Sylvie Billot. A Methodology for the Comparison of Human Judgments With Metrics for Coreference Resolution. HumEval at ACL, May 2022, Dublin, Ireland. pp.16-23, ⟨10.18653/v1/2022.humeval-1.2⟩. ⟨hal-03650294⟩
105 Consultations
92 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More