SONAR EXPRESSIVE: Zero-shot Expressive Speech-to-Speech Translation - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2023

SONAR EXPRESSIVE: Zero-shot Expressive Speech-to-Speech Translation

Résumé

Massively multilingual and multimodal sentence representations like SONAR are usually trained to capture only the meaning of the encoded text or speech. We complement this semantic embedding by a generic speech characteristic embedding which captures the expressive properties of a speech signal. We describe an iterative training procedure which aims to disentangle the semantics and expressive speech properties, and which does not need labeled data. We show the effectiveness of our method on the FLEURS and mEXPRESSO benchmark test sets using multiple metrics which aim to measure the preservation of the meaning and prosody for zero-shot speech-to-speech translation from five languages into English.
Fichier principal
Vignette du fichier
405226870_6969401436468680_1309208683514669610_n.pdf (242.38 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04629427 , version 1 (29-06-2024)

Identifiants

  • HAL Id : hal-04629427 , version 1

Citer

Paul-Ambroise Duquenne, Kevin Heffernan, Alexandre Mourachko, Benoît Sagot, Holger Schwenk. SONAR EXPRESSIVE: Zero-shot Expressive Speech-to-Speech Translation. 2023. ⟨hal-04629427⟩

Collections

INRIA INRIA2
4 Consultations
1 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More