Seeing and Hearing What Has Not Been Said: A multimodal client behavior classifier in Motivational Interviewing with interpretable fusion
Entendre et voir ce qui n'a pas été dit : A classificateur multimodal de comportement de clients d'Interview Motivationnelle avec fusion interprétable
Résumé
Motivational Interviewing (MI) is an approach to therapy that emphasizes collaboration and encourages behavioral change. To evaluate the quality of an MI conversation, client utterances can be classified using the MISC code as either Change Talk (CT), Sustain Talk (ST), or Follow/Neutral (F/N). The proportion of CT in an MI conversation positively correlates with therapy outcomes, making accurate classification of client utterances essential. This paper presents a classifier that accurately distinguishes between the three MISC classes (CT, ST, and F/N), leveraging multimodal features such as text, prosody, and facial expressivity. We annotate the publicly available AnnoMI dataset to train our model to collect multimodal information. Furthermore, we identify the modality that contributes most to the decision-making process, providing valuable insights into the interplay of different modalities during an MI conversation.
Mots clés
Motivational Interviewing
Client Behavior
Multimodal Behavior
Facial Expressions
Long Short-term Memory
Multiple Modalities
Fusion Method
Linguistic Features
Embedding Dimension
Classification Power
Original Transcripts
Multimodal Model
Previous Sentence
Action Units
Self-attention Layer
Backchannel
Final Embedding
BERT Model
Facial Action Units
Text Preprocessing
AdamW Optimizer
Text Modality
Fusion Architecture
Spoken Text
Head Pose
multimodality
interpretable
| Origine | Fichiers produits par l'(les) auteur(s) |
|---|