Multimodal generation of upper-facial and head gestures with a Transformer Network using speech and text - Archive ouverte HAL Access content directly
Preprints, Working Papers, ... Year : 2021

Multimodal generation of upper-facial and head gestures with a Transformer Network using speech and text

Abstract

We propose a semantically-aware speech driven method to generate expressive and natural upper-facial and head motion for Embodied Conversational Agents (ECA). In this work, we tackle two key challenges: produce natural and continuous head motion and upper-facial gestures. We propose a model that generates gestures based on multimodal input features: the first modality is text, and the second one is speech prosody. Our model makes use of Transformers and Convolutions to map the multimodal features that correspond to an utterance to continuous eyebrows and head gestures. We conduct subjective and objective evaluations to validate our approach.

Dates and versions

hal-03570955 , version 1 (13-02-2022)

Identifiers

Cite

Mireille Fares, Catherine Pelachaud, Nicolas Obin. Multimodal generation of upper-facial and head gestures with a Transformer Network using speech and text. 2021. ⟨hal-03570955⟩
77 View
0 Download

Altmetric

Share

Gmail Facebook X LinkedIn More