Multi-element protocol on IR experiments stability: Application to the TREC-COVID test collection ⋆ - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Multi-element protocol on IR experiments stability: Application to the TREC-COVID test collection ⋆

Résumé

The evaluation of information retrieval systems is performed using test collections. The classical Cranfield evaluation paradigm is defined on one fixed corpus of documents and topics. Following this paradigm, several systems can only be compared over the same test collections (documents, topics, assessments). In this work, we explore in a systematic way the impact of similarity of test collections on the comparability of the experiments: characterizing the minimal changes between the collections upon which the performance of IR system evaluated can be compared. To do that, we create pair instances of sub-test collections from one reference collection with controlled overlapping elements, and we compare the Ranking of Systems (RoS) of a defined list of IR systems. We can then compute the probability that the RoS are the same across the sub-test collections. We experiment with our framework proposed on the TREC-COVID collections, and two of our findings show that: a) the ranking of systems, according to the MaP, is very stable even for overlaps smaller than 10% for documents, relevance assessments and positive relevance assessments sub-collections, and b) stability is not ensured for MaP, Rprec, Bpref and ndcg evaluation measures even when considering large overlap for the topics.
Fichier principal
Vignette du fichier
CIRCLE_2022_paper_8.pdf (1.12 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03719613 , version 1 (11-07-2022)

Identifiants

  • HAL Id : hal-03719613 , version 1

Citer

Gabriela Gonzalez-Saez, Philippe Mulhem, Lorraine Goeuriot, Petra Galuščáková. Multi-element protocol on IR experiments stability: Application to the TREC-COVID test collection ⋆. CIRCLE (Joint Conference of the Information Retrieval Communities in Europe), Jul 2022, Samatan, France. ⟨hal-03719613⟩
51 Consultations
17 Téléchargements

Partager

Gmail Facebook X LinkedIn More