Learning Term Discrimination - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2020

Learning Term Discrimination

Résumé

Document indexing is a key component for efficient information retrieval (IR). After preprocessing steps such as stemming and stopword removal, document indexes usually store term-frequencies (tf). Along with tf (that only reflects the importance of a term in a document), traditional IR models use term discrimination values (TDVs) such as inverse document frequency (idf) to favor discriminative terms during retrieval. In this work, we propose to learn TDVs for document indexing with shallow neural networks that approximate traditional IR ranking functions such as TF-IDF and BM25. Our proposal outperforms, both in terms of nDCG and recall, traditional approaches, even with few positively labelled querydocument pairs as learning data. Our learned TDVs, when used to filter out terms of the vocabulary that have zero discrimination value, allow to both significantly lower the memory footprint of the inverted index and speed up the retrieval process (BM25 is up to 3 times faster), without degrading retrieval quality.
Fichier principal
Vignette du fichier
SIGIR2020_for_HAL.pdf (455.41 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03024756 , version 1 (26-11-2020)

Identifiants

Citer

Jibril Frej, Philippe Mulhem, Didier Schwab, Jean-Pierre Chevallet. Learning Term Discrimination. 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020), Jul 2020, Xi'an, China. pp.1993-1996, ⟨10.1145/3397271.3401211⟩. ⟨hal-03024756⟩
28 Consultations
112 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More