Communication Dans Un Congrès Année : 2026

Format Matters: A Critical Evaluation of Output Formats for Prompting LLMs in SLU and NER

Résumé

Output format is often an unreported factor in LLM evaluations for structured NLP tasks such as Slot Filling or Named Entity Recognition. This work proposes to explore the impact of the output structured format generated by LLMs. We show that measured performance and reliability depend on the requested format (JSON, XML or inline Key-Values). A study is performed across four SLU and three NER benchmarks, considering 13 instruction-tuned open-weight LLMs, using standardized and open-source prompts and parsers. This format-specific evaluation reveals statistically significant swings of 2-46 F1 points, depending on model and dataset. Additionally, we propose a lightweight selection procedure to determine the best format per model-dataset combination using only a small development slice; thus reducing trial-and-error in practice.

Fichier principal
Vignette du fichier
main.pdf (325.84 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
licence

Dates et versions

hal-05546569 , version 1 (11-03-2026)

Licence

Identifiants

  • HAL Id : hal-05546569 , version 1

Citer

Pierre Lepagnol, Sahar Ghannay, Thomas Gerald, Christophe Servan, Sophie Rosset. Format Matters: A Critical Evaluation of Output Formats for Prompting LLMs in SLU and NER. The Fifteenth biennial Language Resources and Evaluation Conference (LREC 2026), May 2026, Palma de Majorque, Spain. ⟨hal-05546569⟩
469 Consultations
274 Téléchargements

Partager

  • More