Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition

Résumé

Decades of research indicate that emotion recognition is more effective when drawing information from multiple modalities. But what if some modalities are sometimes missing? To address this problem, we propose a novel Transformer-based architecture for recognizing valence and arousal in a time-continuous manner even with missing input modalities. We use a coupling of cross-attention and self-attention mechanisms to emphasize relationships between modalities during time and enhance the learning process on weak salient inputs. Experimental results on the Ulm-TSST dataset show that our model exhibits an improvement of the concordance correlation coefficient evaluation of 37% when predicting arousal values and 30% when predicting valence values, compared to a late-fusion baseline approach.
Fichier principal
Vignette du fichier
EmoReconMissMod.pdf (200.93 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04287765 , version 1 (15-11-2023)

Identifiants

Citer

Juan Vazquez-Rodriguez, Grégoire Lefebvre, Julien Cumin, James L. Crowley. Accommodating Missing Modalities in Time-Continuous Multimodal Emotion Recognition. Affective Computing and Intelligent Interaction (ACII), Sep 2023, Cambridge (MA), United States. ⟨hal-04287765⟩
20 Consultations
16 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More