Transfer Learning from Whisper for Microscopic Intelligibility Prediction - Archive ouverte HAL
Communication Dans Un Congrès Année : 2024

Transfer Learning from Whisper for Microscopic Intelligibility Prediction

Résumé

Macroscopic intelligibility models predict the expected human word-error-rate for a given speech-in-noise stimulus. In contrast, microscopic intelligibility models aim to make fine-grained predictions about listeners' perception, e.g. predicting phonetic or lexical responses. State-of-the-art macroscopic models use transfer learning from large scale deep learning models for speech processing, whereas such methods have rarely been used for microscopic modeling. In this paper, we study the use of transfer learning from Whisper, a state-of-the-art deep learning model for automatic speech recognition, for microscopic intelligibility prediction at the level of lexical responses. Our method outperforms the considered baselines, even in a zero-shot setup, and yields a relative improvement of up to 66% when fine-tuned to predict listeners' responses. Our results showcase the promise of large scale deep learning based methods for microscopic intelligibility prediction.
Fichier principal
Vignette du fichier
best24_interspeech.pdf (400.59 Ko) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-04683361 , version 1 (04-09-2024)

Identifiants

Citer

Paul Best, Santiago Cuervo, Ricard Marxer. Transfer Learning from Whisper for Microscopic Intelligibility Prediction. Interspeech 2024, Sep 2024, Kos, Greece. pp.3839-3843, ⟨10.21437/Interspeech.2024-2258⟩. ⟨hal-04683361⟩
16 Consultations
4 Téléchargements

Altmetric

Partager

More