InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation

Résumé

Assessing the quality of natural language generation (NLG) systems through human annotation is very expensive. Additionally, human annotation campaigns are time-consuming and include non-reusable human labour. In practice, researchers rely on automatic metrics as a proxy of quality. In the last decade, many string-based metrics (e.g., BLEU or ROUGE) have been introduced. However, such metrics usually rely on exact matches and thus, do not robustly handle synonyms. In this paper, we introduce InfoLM a family of untrained metrics that can be viewed as a string-based metric that addresses the aforementioned flaws thanks to a pre-trained masked language model. This family of metrics also makes use of information measures allowing the possibility to adapt InfoLM to different evaluation criteria. Using direct assessment, we demonstrate that InfoLM achieves statistically significant improvement and two figure correlation gains in many configurations compared to existing metrics on both summarization and data2text generation tasks.

Dates et versions

hal-04575362 , version 1 (14-05-2024)

Identifiants

Citer

Pierre Jean A. Colombo, Chloé Clavel, Pablo Piantanida. InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation. AAAI : Technical Track on Speech and Natural Language Processing, Feb 2022, Online (Guadalajara), Mexico. pp.10554-10562, ⟨10.1609/aaai.v36i10.21299⟩. ⟨hal-04575362⟩
12 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More