ENHANCING EXPRESSIVITY TRANSFER IN TEXTLESS SPEECH-TO-SPEECH TRANSLATION - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

ENHANCING EXPRESSIVITY TRANSFER IN TEXTLESS SPEECH-TO-SPEECH TRANSLATION

Résumé

Textless speech-to-speech translation systems are rapidly advancing, thanks to the integration of self-supervised learning techniques. However, existing state-of-the-art systems fall short when it comes to capturing and transferring expressivity accurately across different languages. Expressivity plays a vital role in conveying emotions, nuances, and cultural subtleties, thereby enhancing communication across diverse languages. To address this issue this study presents a novel method that operates at the discrete speech unit level and leverages multilingual emotion embeddings to capture language-agnostic information. Specifically, we demonstrate how these embeddings can be used to effectively predict the pitch and duration of speech units in the target language. Through objective and subjective experiments conducted on a French-to-English translation task, our findings highlight the superior expressivity transfer achieved by our approach compared to current state-of-the-art systems.
Fichier principal
Vignette du fichier
main.pdf (352.05 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04228410 , version 1 (10-10-2023)

Licence

Copyright (Tous droits réservés)

Identifiants

Citer

Jarod Duret, Benjamin O'Brien, Yannick Estève, Titouan Parcollet. ENHANCING EXPRESSIVITY TRANSFER IN TEXTLESS SPEECH-TO-SPEECH TRANSLATION. ASRU, Dec 2023, Taipei, France. ⟨hal-04228410⟩

Collections

UNIV-AVIGNON LIA
45 Consultations
22 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More