Designing a virtual patient dialogue system based on terminology-rich resources: challenges and evaluation - Archive ouverte HAL
Article Dans Une Revue Natural Language Engineering Année : 2019

Designing a virtual patient dialogue system based on terminology-rich resources: challenges and evaluation

Résumé

Virtual patient software allows health professionals to practice their skills by interacting with tools simulating clinical scenarios. A natural language dialogue system can provide natural interaction for medical history taking. However, the large number of concepts and terms in the medical domain makes the creation of such a system a demanding task. We designed a dialogue system that stands out from current research by its ability to handle a wide variety of medical specialties and clinical cases. To address the task, we designed a patient record model, a knowledge model for the task, and a termino-ontological model that hosts structured thesauri with linguistic, terminological and ontological knowledge. We used a frame- and rule-based approach and terminology-rich resources to handle the medical dialogue. This work focuses on the termino-ontological model, the challenges involved and how the system manages resources for the French language. We adopted a comprehensive approach to collect terms and ontological knowledge, and dictionaries of affixes, synonyms and derivational variants. Resources include domain lists containing over 161,000 ters, and dictionaries with over 959,000 word/concept entries. We assessed our approach by having 71 participants (39 medical doctors and 32 non-medical evaluators) interact with the system and use 35 cases from 18 specialities. We conducted a quantitative evaluation of all components by analysing interaction logs (11,834turns). Natural language understanding achieved an F-measure of 95.8 per cent. Dialogue management provided on average 74.3 (±9.5) per cent of correct answers. We performed a qualitative evaluation by collecting 171 five-point Likert scale questionnaires. All evaluated aspects obtained mean scores above the Likert mid-scale point. We analysed the vocabulary coverage with regard to unseen cases: the system covered 97.8 per cent of theirterms. Evaluations showed that the system achieved high vocabulary coverage on unseen cases and was assessed as relevant for the task.
Fichier principal
Vignette du fichier
pg_nle_final.pdf (5.38 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-02358021 , version 1 (27-07-2021)

Identifiants

Citer

Leonardo Campillos-Llanos, Catherine Thomas, Eric Bilinski, Pierre Zweigenbaum, Sophie Rosset. Designing a virtual patient dialogue system based on terminology-rich resources: challenges and evaluation. Natural Language Engineering, 2019, 26 (2), pp. 183 - 220. ⟨10.1017/S1351324919000329⟩. ⟨hal-02358021⟩
138 Consultations
130 Téléchargements

Altmetric

Partager

More