Learning Multilingual Expressive Speech Representation for Prosody Prediction without Parallel Data - Archive ouverte HAL
Communication Dans Un Congrès Année : 2023

Learning Multilingual Expressive Speech Representation for Prosody Prediction without Parallel Data

Résumé

We propose a method for speech-to-speech emotionpreserving translation that operates at the level of discrete speech units. Our approach relies on the use of multilingual emotion embedding that can capture affective information in a language-independent manner. We show that this embedding can be used to predict the pitch and duration of speech units in a target language, allowing us to resynthesize the source speech signal with the same emotional content. We evaluate our approach to English and French speech signals and show that it outperforms a baseline method that does not use emotional information, including when the emotion embedding is extracted from a different language. Even if this preliminary study does not address directly the machine translation issue, our results demonstrate the effectiveness of our approach for cross-lingual emotion preservation in the context of speech resynthesis.
Fichier principal
Vignette du fichier
main.pdf (289.21 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04144850 , version 1 (28-06-2023)

Identifiants

Citer

Jarod Duret, Titouan Parcollet, Yannick Estève. Learning Multilingual Expressive Speech Representation for Prosody Prediction without Parallel Data. Speech Synthesis Workshop (SSW), Aug 2023, Grenoble, France. ⟨hal-04144850⟩

Collections

UNIV-AVIGNON LIA
24 Consultations
133 Téléchargements

Altmetric

Partager

More