Video-based multimodal spontaneous emotion recognition using facial expressions and physiological signals - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Video-based multimodal spontaneous emotion recognition using facial expressions and physiological signals

Résumé

Human's affective state recognition remains a challenging topic due to the complexity of emotions, which involves experiential, behavioral, and physiological elements. Since it is difficult to comprehensively describe emotion in terms of single modalities, recent studies have focused on fusion strategy to exploit the complementarity of multimodal signals. In this article, we study the feasibility of fusing facial expressions with physiological cues on human emotion recognition accuracy. The contributions of this work are threefold: 1) We propose a new spatiotemporal network for facial expression recognition using a 3D squeeze and exitation based 3D Xception architecture (squeeze and exitation Xception network). 2) We adopt the first multiple modalities fusion using single input source which, to the best of our knowledge, no existing multimodal emotion recognition system has attempted to identify emotional state from only facial videos using facial expressions and physiological signals features. 3) We compare the performance of the unimodal approach using only facial expressions or physiological data, to multimodal systems fusing facial expressions with video-based physiological cues. In our experiments, physiological signals such as the iPPG signal and features of heart rate variability measured remotely using the imaging photoplethysmography (iPPG) method are used. The preliminary results show that the multimodal fusion model improves the accuracy of emotion recognition, and merging facial expressions features with iPPG signal gives the best accuracy with 71.90 %.
Fichier principal
Vignette du fichier
Ouzar_Video-Based_Multimodal_Spontaneous_Emotion_Recognition_Using_Facial_Expressions_and_Physiological_CVPRW_2022_paper.pdf (991.64 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03790802 , version 1 (28-09-2022)

Identifiants

Citer

Yassine Ouzar, Frédéric Bousefsaf, Djamaleddine Djeldjli, Choubeila Maaoui. Video-based multimodal spontaneous emotion recognition using facial expressions and physiological signals. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Jun 2022, New Orleans, United States. pp.2459-2468, ⟨10.1109/CVPRW56347.2022.00275⟩. ⟨hal-03790802⟩
30 Consultations
119 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More