Audio-video fusion with double attention for multimodal emotion recognition - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Audio-video fusion with double attention for multimodal emotion recognition

Résumé

Recently, the multimodal emotion recognition has become a hot topic of research, within the affective computing community, due to its robust performances. In this paper, we propose to analyze emotions in an end-to-end manner based on various convolutional neural networks (CNN) architectures and attention mechanisms. Specifically, we develop a new framework that integrates the spatial and temporal attention into a visual 3D-CNN and temporal attention into an audio 2D-CNN in order to capture the intra-modal features characteristics. Further, the system is extended with an audio-video cross-attention fusion approach that effectively exploits the relationship across the two modalities. The proposed method achieves 87.89% of accuracy on RAVDESS dataset. When compared with state-of-the art methods our system demonstrates accuracy gains of more than 1.89%.
Fichier non déposé

Dates et versions

hal-03937090 , version 1 (13-01-2023)

Identifiants

Citer

Bogdan Mocanu, Ruxandra Tapu. Audio-video fusion with double attention for multimodal emotion recognition. 2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), Jun 2022, Nafplio, Greece. pp.1-5, ⟨10.1109/IVMSP54334.2022.9816349⟩. ⟨hal-03937090⟩
10 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More