The structure of unseen trigrams and its application to language models: A first investigation - Archive ouverte HAL
Communication Dans Un Congrès Année : 2010

The structure of unseen trigrams and its application to language models: A first investigation

Résumé

In a series of preparatory experiments in 4 languages on subsets of the Europarl corpus, we show that a large number of unseen trigrams can be reconstructed by proportional analogy with trigrams having the lowest frequencies. We derive a very simple smoothing scheme from this empirical result and show that it outperforms Good-Turing and Kneser-Ney smoothing schemes on trigrams models in all 11 languages on the common multilingual part of the Europarl corpus, except Finnish.
Fichier non déposé

Dates et versions

hal-00606385 , version 1 (06-07-2011)

Identifiants

Citer

Yves Lepage, Julien Gosme, Adrien Lardilleux. The structure of unseen trigrams and its application to language models: A first investigation. The 4th International Universal Communication Symposium (IUCS 2010), Oct 2010, Beijing, China. pp.273-280, ⟨10.1109/IUCS.2010.5666011⟩. ⟨hal-00606385⟩
62 Consultations
0 Téléchargements

Altmetric

Partager

More