Development of a Silent Speech Interface Driven by Ultrasound and Optical Images of the Tongue and Lips - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Speech Communication Année : 2010

Development of a Silent Speech Interface Driven by Ultrasound and Optical Images of the Tongue and Lips

Thomas Hueber
Connectez-vous pour contacter l'auteur
Elie-Laurent Benaroya
Gérard Chollet
Bruce Denby
  • Fonction : Auteur
Gérard Dreyfus
  • Fonction : Auteur
Maureen Stone
  • Fonction : Auteur

Résumé

This article presents a segmental vocoder driven by ultrasound and optical images (standard CCD camera) of the tongue and lips for a "silent speech interface" application, usable either by a laryngectomized patient or for silent communication. The system is built around an audio-visual dictionary which associates visual to acoustic observations for each phonetic class. Visual features are extracted from ultrasound images of the tongue and from video images of the lips using a PCA-based image coding technique. Visual observations of each phonetic class are modeled by continuous HMMs. The system then combines a phone recognition stage with corpus-based synthesis. In the recognition stage, the visual HMMs are used to identify phonetic targets in a sequence of visual features. In the synthesis stage, these phonetic targets constrain the dictionary search for the sequence of diphones that maximizes similarity to the input test data in the visual space, subject to a concatenation cost in the acoustic domain. A prosody template is extracted from the training corpus, and the final speech waveform is generated using "Harmonic plus Noise Model" concatenative synthesis techniques. Experimental results are based on an audiovisual database containing one hour of continuous speech from each of two speakers.

Mots clés

Fichier principal
Vignette du fichier
PEER_stage2_10.1016%2Fj.specom.2009.11.004.pdf (3.42 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-00616228 , version 1 (20-08-2011)

Identifiants

Citer

Thomas Hueber, Elie-Laurent Benaroya, Gérard Chollet, Bruce Denby, Gérard Dreyfus, et al.. Development of a Silent Speech Interface Driven by Ultrasound and Optical Images of the Tongue and Lips. Speech Communication, 2010, 52 (4), pp.288. ⟨10.1016/j.specom.2009.11.004⟩. ⟨hal-00616228⟩

Collections

PEER
99 Consultations
566 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More