Mapping the Landscape of Dialogue Research: A Meta-Analysis of ACL Anthology 2024
Résumé
Conversation is a foundational yet methodologically challenging object of study in computational linguistics. While dialogue has long been central to human communication, its multimodal, interactive, and context-dependent nature complicates both data collection and evaluation. In this meta-analysis, we examine how conversation is currently conceptualised and studied within the NLP/CL community by analysing dialogue-related papers published in 2024 across major venues indexed in the ACL Anthology. Using metadata retrieved via the ACL Anthology API and manual annotation, we identify 445 papers addressing conversational phenomena. We analyse the conversational tasks studied, the corpora employed, and the evaluation methods used. Our findings show a strong dominance of task-oriented dialogue systems, with widespread reliance on a limited set of benchmark datasets and on automatic reference-based evaluation metrics. We highlight recurring issues, including insufficient corpus documentation, the use of scripted or synthetic dialogue as a proxy for natural conversation, and evaluation practices that inadequately capture conversational quality. We argue for the need to make task-specific expectations of conversation explicit and to develop evaluation frameworks that better reflect the richness and diversity of conversational interaction.
Domaines
| Origine | Fichiers éditeurs autorisés sur une archive ouverte |
|---|---|
| Licence |