Communication Dans Un Congrès Année : 2024

FastLips: an End-to-End Audiovisual Text-to-Speech System with Lip Features Prediction for Virtual Avatars

Résumé

In this paper, we introduce FastLips, an end-to-end neural model designed to generate speech and co-verbal facial movements from text, animating a virtual avatar. Based on the Fast-Speech2 Text-to-Speech model, FastLips integrates an audiovisual Transformer-based encoder with distinct audio and visual neural decoders. This model combines audiovisual representations computed by the shared encoder with asynchronous generation of audio and visual features. Furthermore, we enhance the model with explicit predictors of lip aperture and spreading, adapted from prosodic FastSpeech2's variance adaptor. The proposed model generates mel-spectrograms and facial features (head, eyes, jaw and lip movements) to drive the virtual avatar's action units. In our evaluation, we compare FastLips with a baseline audiovisual-Tacotron2, demonstrating the advantages of the FastSpeech2 architecture for lip generation. This benefit becomes particularly prominent when implementing explicit lip prediction.

Fichier principal
Vignette du fichier
lenglet24_interspeech.pdf (645.36 Ko) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte
Licence

Dates et versions

hal-04683663 , version 1 (02-09-2024)

Licence

Identifiants

Citer

Martin Lenglet, Olivier Perrotin, Gérard Bailly. FastLips: an End-to-End Audiovisual Text-to-Speech System with Lip Features Prediction for Virtual Avatars. Interspeech 2024 - 25th Annual Conference of the International Speech Communication Association, Sep 2024, Kos, Greece. pp.3450-3454, ⟨10.21437/Interspeech.2024-462⟩. ⟨hal-04683663⟩
495 Consultations
470 Téléchargements

Altmetric

Partager

  • More