Validity, Agreement, Consensuality and Annotated Data Quality - Archive ouverte HAL
Poster De Conférence Année : 2022

Validity, Agreement, Consensuality and Annotated Data Quality

Résumé

Reference annotated (or gold-standard) datasets are required for various common tasks such as training for machine learning systems or system validation. They are necessary to analyse or compare occurrences or items annotated by experts, or to compare objects resulting from any computational process to objects annotated (selected and characterized) by experts. But, even if reference annotated gold-standard corpora are required, their production is known as a difficult problem, from both a theoretical and practical point of view. Many studies devoted to these issues conclude that multi-annotation is most of the time a necessity. Measuring the inter-annotator agreement, which is required to check the reliability of data and the reproducibility of an annotation task, and thus to establish a gold standard, is another thorny problem. Fine analysis of available metrics for this specific task then becomes essential. Our work is part of this effort and more precisely focuses on several problems, which are rarely discussed, although they are intrinsically linked with the interpretation and the evaluation of metrics. In particular, we focus here on the complex relations between agreement and reference (of which agreement among annotators is supposed to be an indicator), and the emergence of a consensus. We also introduce the notion of consensuality as another relevant indicator.
Fichier non déposé

Dates et versions

hal-03784912 , version 1 (23-09-2022)

Identifiants

  • HAL Id : hal-03784912 , version 1

Citer

Anaëlle Baledent, Yann Mathet, Antoine Widlöcher, Christophe Couronne, Jean-Luc Manguin. Validity, Agreement, Consensuality and Annotated Data Quality. 13th Conference on Language Resources and Evaluation (LREC 2022), Jun 2022, Marseille, France. Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022). ⟨hal-03784912⟩
39 Consultations
0 Téléchargements

Partager

More