Optimising Multiple Metrics with MERT - Archive ouverte HAL
Article Dans Une Revue The Prague Bulletin of Mathematical Linguistics Année : 2011

Optimising Multiple Metrics with MERT

Résumé

The main metric used for SMT systems evaluation an optimisation is BLEU score but this metric is questioned about its relevance to human evaluation. Some other metrics already exist but none of them are in perfect harmony with human evaluation. On the other hand, most evaluations use multiple metrics (BLEU, TER, METEOR, etc.). Systems can optimise toward other metrics than BLEU. But optimisation with other metrics tends to decrease BLEU score. As Machine Translation evaluations still use BLEU as main metric, it is important to min-imise the decrease of BLEU. We propose to optimise toward a metric combination like BLEU-TER. This proposition includes two new open source scorers for MERT, the SMT optimisation tool. The first one is a TER scorer that allows us to optimise toward TER; the second one is a combination scorer. The latter one enables the combination of two or more metrics for the optimisation process. This paper also presents some experiments on the MERT optimisation in the Statistical Machine Translation system Moses with the TER and the BLEU metrics and some metric combinations .
Fichier principal
Vignette du fichier
art-servan-schwenk.pdf (140.02 Ko) Télécharger le fichier
Origine Publication financée par une institution
Loading...

Dates et versions

hal-01157949 , version 1 (29-05-2015)

Licence

Copyright (Tous droits réservés)

Identifiants

  • HAL Id : hal-01157949 , version 1

Citer

Christophe Servan, Holger Schwenk. Optimising Multiple Metrics with MERT. The Prague Bulletin of Mathematical Linguistics, 2011, 96, pp.109. ⟨hal-01157949⟩

Relations

294 Consultations
272 Téléchargements

Partager

More