Evaluating Multilingual Small Language Models in Medical and Educational Settings
Abstract
Large Language Models are becoming more performant in different tasks, such as summarizing, chatbot dialogue or question answering. Moreover, LLMs are still not reliable enough to be used as plug-and-play tools in medical or educational contexts, especially in a multilingual context (which reflects the most a real world usage of these tools). Multilingual LLMs are not equally performant on all languages, their performance being lower on languages other than English. Furthermore, LLMs are not adapted to understand the complexity of the medical knowledge, nor to be learning assistants for young students. We evaluated the multilingual Small Language Models (less than 3 billion parameters) performance on QA tasks in three languages, English, French, and Romanian, a less resourced language. We present results on the model’s capabilities in answering medical research questions of different levels of complexity in terms of technical knowledge and reasoning, and assisting 11 year-old schoolchildren with their homework.