Communication Dans Un Congrès Année : 2025

Evaluation of Neural Machine Translation on a Chinese–Vietnamese Parallel Corpus Using Automatic Metrics and Human Annotations (MQM)

Résumé

This project aims to evaluate the quality of neural machine translations or NMT (Barbin 2020) from Vietnamese to Chinese in specialized fields (medicine, gastronomy, biblical), by crossing automatic metrics (eg. BLEU, CHRF) with human annotations (Snover and Dorr 2006) following the MQM (Multidimensional Quality Metrics) framework. The Chinese (Traditional characters)–Vietnamese parallel corpus (in TMX format), extracted from the OPUS platform, was cleaned, segmented into 100 segments, and annotated in a CSV file with the following columns: system, doc, doc_id, seg_id, rater, source, target, category, and severity. A) Reference example: Vả , đất là vô_hình và trống_không , sự mờ tối ở trên mặt vực ; Thần_Ðức_Chúa_Trời vận_hành trên mặt_nước . 地是 空虛 混沌 . 淵面 黑暗 . 神 的 靈運 行在 水面上 Ðức_Chúa_Trời phán rằng : Phải có sự sáng ; thì_có sự sáng . 神說 、 要 有 光 、 就 有 了 光 B) Examples of candidates: Each segment was translated by two separate systems, for example: 1) a standard engine (Google translation): Vả , đất là vô_hình và trống_không , sự mờ tối ở trên mặt vực ; Thần_Ðức_Chúa_Trời vận_hành trên mặt_nước . Ðức_Chúa_Trời phán rằng : Phải có sự sáng ; thì_có sự sáng . 地是空虛混沌,淵面黑暗。神的靈運行在水面上。。 神說:「要有光!」就有了光。 2) a large language model (ChatGPT 4.0). 
Vả , đất là vô_hình và trống_không , sự mờ tối ở trên mặt vực ; Thần_Ðức_Chúa_Trời vận_hành trên mặt_nước .
Ðức_Chúa_Trời phán rằng : Phải có sự sáng ; thì_có sự sáng .
地是空虛混沌,淵面黑暗;上帝的靈運行在水面上。 上帝說:「要有光」,就有了光。 Automatic evaluation of translations was conducted using four complementary metrics — BLEU (Papineni et al. 2002), TER (https://github.com/fenix01/traduction), chrF (Popovic 2015), and BERTScore (https://github.com/Tiiiger/bert_score) — to compare the performance of the reference translation, Google Translate, and ChatGPT. The results show that Google Translate outperforms ChatGPT across all metrics. This can likely be attributed to Google’s translation engine being more optimized for Chinese, benefiting from extensive domain-specific data and targeted fine-tuning. ChatGPT, while slightly behind in lexical accuracy, remains competitive in terms of semantic similarity, as indicated by its high BERTScore. This suggests that while it may deviate from the reference in wording, it often captures the intended meaning well. To deepen the analysis, we compare these automatic scores with human evaluation conducted using the MQM framework (via Marot). This comparison aims to address key questions: Which automatic metrics align most closely with human judgment? In what contexts do these metrics fail to reflect the quality perceived by professional translators? What do discrepancies between human evaluation and automatic scoring reveal about the limitations of current automated evaluation methods? Beyond a simple comparison, we reflect on the broader implications for translation assessment. Can translation quality really be captured by a set of numerical scores? To what extent is human expertise essential for interpreting or calibrating these metrics? In cases of mismatch, should the error be attributed to the translation system, or to the metric itself? Furthermore, the project includes the pretraining and fine-tuning of a translation model on a specialized corpus — for instance, using mBART50 or mBART-large-50 — selected based on the specific characteristics of the task. We have already developed an initial version of the model using this Chinese–Vietnamese parallel corpus, trained and fine-tuned by ourselves with mBART. For example: Vả, đất là vô hình và trống không, sự mờ tối ở trên mặt vực; Thần Đức Chúa Trời vận hành trên mặt nước. Đức Chúa Trời phán rằng: Phải có sự sáng; thì có sự sáng. 地 是 無 形 的 、 空 的 、 幽 暗 在 外 面 . 神 的 靈 力 在 水 面 上 行 神 說 、 有 光 、 就 是 光 The objective is to build a tailored model optimized for this specific language pair (Vietnamese–Chinese) and domain (eg. biblical and Sino-Vietnamese texts), while also evaluating the effectiveness of our automatic evaluation tools. This exploratory work not only improves translation quality through domain adaptation, but also lays the groundwork for future experiments involving larger or multilingual corpora — including French, Korean, English, Japanese, and more.

Fichier non déposé

Dates et versions

hal-05113434 , version 1 (15-06-2025)

Identifiants

  • HAL Id : hal-05113434 , version 1

Citer

Lian Chen, Huy Linh Dao. Evaluation of Neural Machine Translation on a Chinese–Vietnamese Parallel Corpus Using Automatic Metrics and Human Annotations (MQM). The 38th Paris Meeting on East Asian Linguistics (JLAO 38), CHIRKOVA Katia; WIEBUSCH Thekla; DAO Huy-Linh; CHEN Lian (陈恋); DONG Yao (董瑶); YIN Yuanhao (殷元昊), Jul 2025, Paris, France. ⟨hal-05113434⟩
296 Consultations
0 Téléchargements

Partager

  • More