Direct vs. indirect evaluation of distributional thesauri - Archive ouverte HAL Access content directly
Conference Papers Year : 2016

Direct vs. indirect evaluation of distributional thesauri


With the success of word embedding methods in various Natural Language Processing tasks, all the fields of distributional semantics have experienced a renewed interest. Beside the famous word2vec, recent studies have presented efficient techniques to build distributional thesaurus; in particular, Claveau et al. (2014) have already shown that Information Retrieval (IR) tools and concepts can be successfully used to build a thesaurus. In this paper, we address the problem of the evaluation of such thesauri or embedding models. Several evaluation scenarii are considered: direct evaluation through reference lexicons and specially crafted datasets, and indirect evaluation through a third party tasks, namely lexical subsitution and Information Retrieval. For this latter task, we adopt the query expansion framework proposed by Claveau and Kijak (2016). Through several experiments, we first show that the recent techniques for building distributional thesaurus outperform the word2vec approach, whatever the evaluation scenario. We also highlight the differences between the evaluation scenarii, which may lead to very different conclusions when comparing distributional models. Last, we study the effect of some parameters of the distributional models on these various evaluation scenarii.
Fichier principal
Vignette du fichier
Claveau_Kijak_IR_COLING2016.pdf (460.61 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-01394739 , version 1 (09-11-2016)


  • HAL Id : hal-01394739 , version 1


Vincent Claveau, Ewa Kijak. Direct vs. indirect evaluation of distributional thesauri. International Conference on Computational Linguistics, COLING, Dec 2016, Osaka, Japan. ⟨hal-01394739⟩
236 View
192 Download


Gmail Mastodon Facebook X LinkedIn More