Direct vs. indirect evaluation of distributional thesauri - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2016

Direct vs. indirect evaluation of distributional thesauri

Résumé

With the success of word embedding methods in various Natural Language Processing tasks, all the fields of distributional semantics have experienced a renewed interest. Beside the famous word2vec, recent studies have presented efficient techniques to build distributional thesaurus; in particular, Claveau et al. (2014) have already shown that Information Retrieval (IR) tools and concepts can be successfully used to build a thesaurus. In this paper, we address the problem of the evaluation of such thesauri or embedding models. Several evaluation scenarii are considered: direct evaluation through reference lexicons and specially crafted datasets, and indirect evaluation through a third party tasks, namely lexical subsitution and Information Retrieval. For this latter task, we adopt the query expansion framework proposed by Claveau and Kijak (2016). Through several experiments, we first show that the recent techniques for building distributional thesaurus outperform the word2vec approach, whatever the evaluation scenario. We also highlight the differences between the evaluation scenarii, which may lead to very different conclusions when comparing distributional models. Last, we study the effect of some parameters of the distributional models on these various evaluation scenarii.
Fichier principal
Vignette du fichier
Claveau_Kijak_IR_COLING2016.pdf (460.61 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01394739 , version 1 (09-11-2016)

Identifiants

  • HAL Id : hal-01394739 , version 1

Citer

Vincent Claveau, Ewa Kijak. Direct vs. indirect evaluation of distributional thesauri. International Conference on Computational Linguistics, COLING, Dec 2016, Osaka, Japan. ⟨hal-01394739⟩
233 Consultations
165 Téléchargements

Partager

Gmail Facebook X LinkedIn More