Further results on latent discourse models and word embeddings - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Journal of Machine Learning Research Année : 2021

Further results on latent discourse models and word embeddings

Résumé

We discuss some properties of generative models for word embeddings. Namely, (Arora et al., 2016) proposed a latent discourse model implying the concentration of the partition function of the word vectors. This concentration phenomenon led to an asymptotic linear relation between the pointwise mutual information (PMI) of pairs of words and the scalar product of their vectors. Here, we first revisit this concentration phenomenon and prove it under slightly weaker assumptions, for a set of random vectors symmetrically distributed around the origin. Second, we empirically evaluate the relation between PMI and scalar products of word vectors satisfying the concentration property. Our empirical results indicate that, in practice, this relation does not hold with arbitrarily small error. This observation is further supported by two theoretical results: (i) the error cannot be exactly zero because the corresponding shifted PMI matrix cannot be positive semidefinite; (ii) under mild assumptions, there exist pairs of words for which the error cannot be close to zero. We deduce that either natural language does not follow the assumptions of the considered generative model, or the current word vector generation methods do not allow the construction of the hypothesized word embeddings.
Fichier principal
Vignette du fichier
Discussion_generative_model_latent_word_embeddings__Copy_.pdf (986.58 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02983109 , version 1 (29-10-2020)

Identifiants

  • HAL Id : hal-02983109 , version 1

Citer

Sammy Khalife, Douglas S Gonçalves, Youssef Allouah, Leo Liberti. Further results on latent discourse models and word embeddings. Journal of Machine Learning Research, 2021. ⟨hal-02983109⟩
133 Consultations
68 Téléchargements

Partager

Gmail Facebook X LinkedIn More