Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features - Archive ouverte HAL
Communication Dans Un Congrès Année : 2023

Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features

Résumé

This paper explores privacy-compliant group-level emotion recognition "in-the-wild" within the EmotiW Challenge 2023. Group-level emotion recognition can be useful in many fields including social robotics, conversational agents, e-coaching and learning analytics. This research imposes itself using only global features avoiding individual ones, i.e. all features that can be used to identify or track people in videos (facial landmarks, body poses, audio diarization, etc.). The proposed multimodal model is composed of a video and an audio branches with a cross-attention between modalities. The video branch is based on a fine-tuned ViT architecture. The audio branch extracts Mel-spectrograms and feed them through CNN blocks into a transformer encoder. Our training paradigm includes a generated synthetic dataset to increase the sensitivity of our model on facial expression within the image in a data-driven way. The extensive experiments show the significance of our methodology. Our privacy-compliant proposal performs fairly on the EmotiW challenge, with 79.24% and 75.13% of accuracy respectively on validation and test set for the best models. Noticeably, our findings highlight that it is possible to reach this accuracy level with privacy-compliant features using only 5 frames uniformly distributed on the video.
Fichier principal
Vignette du fichier
MgEmoR-pcf-Emotiw2023.pdf (2.81 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04325815 , version 1 (06-12-2023)

Identifiants

Citer

Anderson Augusma, Dominique Vaufreydaz, Frédérique Letué. Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features. ICMI '23: International Conference on Multimodal Interaction, Oct 2023, Paris, France. pp.750-754, ⟨10.1145/3577190.3616546⟩. ⟨hal-04325815⟩
183 Consultations
79 Téléchargements

Altmetric

Partager

More