An experimental analysis of Noise-Contrastive Estimation: the noise distribution matters
Résumé
Noise Contrastive Estimation (NCE) is a learning procedure that is regularly used to train neural language models, since it avoids the computational bottleneck caused by the output softmax. In this paper , we attempt to explain some of the weaknesses of this objective function, and to draw directions for further developments. Experiments on a small task show the issues raised by the unigram noise distribution, and that a context dependent noise distribution, such as the bigram distribution , can solve these issues and provide stable and data-efficient learning.
| Origine | Fichiers éditeurs autorisés sur une archive ouverte |
|---|
Loading...