Semantic and Pragmatic Properties of LLM’s "Hallucinations" in the Medical Field
Résumé
Our study focuses on "hallucinations", language productions generated by Large Language Models (LLM) that do not correspond to the expected output. The term "hallucination" has been questioned by a series of recent publications. In general, the literature on hallucination typology does not approach productions from the point of view of language structures, but from the perspective of the requested tasks, such as machine translation, question answering, summaries, etc. (Bruno et al., 2023; Zhang et al., 2023). In our work we tested the hypothesis whether it is possible to identify language patterns (such as morphosyntactic or semantic patterns) in incorrect generations of medical term paraphrases. The aim of medical paraphrases is to explain and to simplify medical terms and make medical knowledge accessible to the general public (Grabar & Hamon, 2015; Buhnila, 2022). Our two main contributions are: [1] a fine-grained linguistic analysis of LLM "hallucinations" in the context of medical paraphrase generation; [2] our morphosyntactic and semantic analysis of recurring patterns identified in the hallucinations showed the LLM’s tendency to over-generalize or to mimic the human natural repetition habit as a learning process.