The success of Large Language Models (LLM) has led to controversial hypotheses about emergent intelligence beyond language competence, amongst which Theory of Mind (ToM) has become the most debated. While some reported gpt-4 to pass the Sally-Anne test, others reported limitations. Here we aim to validate LLM ability to recognise ToM occurrences in natural human dialogue using the activity of ToM-associated brain regions.

The current analysis uses an existing corpus of 25 participants recorded with functional MRI while carrying conversations in 4 sessions x 6 trials alternating a Human and a Robot interlocutor. The conversation is considered natural as participants are provided with a cover story hiding the actual objective of the experiment to study social interactions, and is known to elicit ToM phenomena. We analysed the resulting 100 hours of transcribed conversations synchronised with fMRI recordings by a bespoke gpt-4-turbo prompt. Each conversation turn from the participant and its interlocutors were tagged as containing (ToM+) or not (ToM-) references to mental states by the LLM. Speakers (2) and ToM category (2) defined the four conditions used in a general linear model analysis of whole-brain response.

Contrast between conversation turns tagged ToM+ versus ToM-by gpt-4 revealed significant activations (p<0.1 FWE-corrected at the cluster-level ) in the left TemporoParietal Junction, Medial PreFrontal Cortex and lateral OrbitoFrontal Cortex, aligning with the implementation of mentalizing mechanisms in the human brain. Our contribution differs from previous work in the ecological nature of the task, which contrasts with studies based on dedicated ToM tests.