Article Dans Une Revue IEEE Transactions on Affective Computing Année : 2025

Exploring emotion expression recognition in older adults interacting with a virtual Coach

Mohamed Amine Hmani
Aymen Mtibaa
  • Fonction : Auteur
  • PersonId : 1038682
Pau Buch-Cardona
  • Fonction : Auteur
Raquel Justo
  • Fonction : Auteur
  • PersonId : 1092997
Terry Amorese
  • Fonction : Auteur
Eduardo González-Fraile
  • Fonction : Auteur
Begoña Fernández-Ruanova
  • Fonction : Auteur
Jofre Tenorio-Laranga
  • Fonction : Auteur
Anna Torp Johansen
  • Fonction : Auteur
Micaela Rodrigues da Silva
  • Fonction : Auteur
Liva Jenny Martinussen
  • Fonction : Auteur
Maria Stylianou Korsnes
  • Fonction : Auteur
Gennaro Cordasco
  • Fonction : Auteur
  • PersonId : 1092998
Anna Esposito
  • Fonction : Auteur
Mounim El Yacoubi
María Inés Torres
  • Fonction : Auteur
  • PersonId : 1335191
Sergio Escalera
  • Fonction : Auteur
  • PersonId : 854688

Résumé

The EMPATHIC project aimed to design an emotionally expressive virtual coach capable of engaging healthy seniors to improve well-being and promote independent aging. In particular, the system's human sensing capabilities allow for the perception of emotional states to provide a personalized experience. This paper outlines the development of the emotion expression recognition module of the virtual coach, encompassing data collection, annotation design, and a first methodological approach, all tailored to the project requirements. With the latter, we investigate the role of various modalities, individually and combined, for discrete emotion expression recognition in this context: speech from audio, and facial expressions, gaze, and head dynamics from video. The collected corpus includes users from Spain, France, and Norway, and was annotated separately for the audio and video channels with distinct emotional labels, allowing for a performance comparison across cultures and label types. Results confirm the informative power of the modalities studied for the emotional categories considered, with multimodal methods generally outperforming others (around 68% accuracy with audio labels and 72-74% with video labels). The findings are expected to contribute to the limited literature on emotion recognition applied to older adults in conversational human-machine interaction, and guide the development of future systems.

Fichier principal
Vignette du fichier
Exploring_Emotion_Expression_Recognition_in_Older_Adults_Interacting_With_a_Virtual_Coach.pdf (5.28 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05280245 , version 1 (24-09-2025)

Licence

Identifiants

Citer

Cristina Palmero, Mikel Develasco, Mohamed Amine Hmani, Aymen Mtibaa, Leila Ben Letaifa, et al.. Exploring emotion expression recognition in older adults interacting with a virtual Coach. IEEE Transactions on Affective Computing, 2025, 16 (3), pp.2303 - 2320. ⟨10.1109/taffc.2025.3558141⟩. ⟨hal-05280245⟩
50 Consultations
144 Téléchargements

Altmetric

Partager

  • More