Identifying Causal Relationships Between Behavior and Local Brain Activity During Natural Conversation - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2020

Identifying Causal Relationships Between Behavior and Local Brain Activity During Natural Conversation

Résumé

Characterizing precisely neurophysiological activity involved in natural conversations remains a major challenge. We explore in this paper the relationship between multimodal conversational behavior and brain activity during natural conversations. This is challenging due to Functional Magnetic Resonance Imaging (fMRI) time resolution and to the diversity of the recorded multimodal signals. We use a unique corpus including localized brain activity and behavior recorded during a fMRI experiment when several participants had natural conversations alternatively with a human and a conversational robot. The corpus includes fMRI responses as well as conversational signals that consist of synchronized raw audio and their transcripts, video and eye-tracking recordings. The proposed approach includes a first step to extract discrete neurophysiological time-series from functionally well defined brain areas, as well as behavioral time-series describing specific behaviors. Then, machine learning models are applied to predict neurophysiological time-series based on the extracted behavioral features. The results show promising prediction scores, and specific causal relationships are found between behaviors and the activity in functional brain areas for both conditions, i.e., human-human and human-robot conversations.
Fichier principal
Vignette du fichier
hmamouche_INTERSPEECH2020.pdf (420.42 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03016922 , version 1 (03-12-2020)

Identifiants

Citer

Youssef Hmamouche, Laurent Prevot, Ochs Magalie, Chaminade Thierry. Identifying Causal Relationships Between Behavior and Local Brain Activity During Natural Conversation. Interspeech 2020, Oct 2020, Shanghai, China. pp.101-105, ⟨10.21437/Interspeech.2020-2074⟩. ⟨hal-03016922⟩
123 Consultations
201 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More