Automatic facial expressions, gaze direction and head movements generation of a virtual agent - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Automatic facial expressions, gaze direction and head movements generation of a virtual agent

Résumé

In this article, we present two models to jointly and automatically generate the head, facial and gaze movements of a virtual agent from acoustic speech features. Two architectures are explored: a Generative Adversarial Network and an Adversarial Encoder-Decoder. Head movements and gaze orientation are generated as 3D coordinates, while facial expressions are generated using action units based on the facial action coding system. A large corpus of almost 4 hours of videos, involving 89 different speakers is used to train our models. We extract the speech and visual features automatically from these videos using existing tools. The evaluation of these models is conducted objectively with measures such as density evaluation and a visualisation from PCA reduction, as well as subjectively through a users perceptive study. Our proposed methodology shows that on 15 seconds sequences, encoder-decoder architecture drastically improves the perception of generated behaviours in two criteria: the coordination with speech and the naturalness. Our code can be found in : https://github.com/aldelb/nonverbal-behaviours-generation.
Fichier principal
Vignette du fichier
Automatic facial expressions, gaze direction and head movements generation of a virtual agent.pdf (675.02 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03794365 , version 1 (03-10-2022)

Identifiants

Citer

Alice Delbosc, Magalie Ochs, Stéphane Ayache. Automatic facial expressions, gaze direction and head movements generation of a virtual agent. GENEA : GENERATION AND EVALUATION OF NON-VERBAL BEHAVIOUR FOR EMBODIED AGENTS, Nov 2022, Bengaluru, India. ⟨10.1145/3536220.3558806⟩. ⟨hal-03794365⟩
42 Consultations
89 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More