Communication Dans Un Congrès Année : 2025

CBF-AFA: Chunk-Based Multi-SSL Fusion for Automatic Fluency Assessment

Résumé

Automatic fluency assessment (AFA) remains challenging, particularly in capturing speech rhythm, pauses, and disfluencies in non-native speakers. We introduce a chunk-based approach integrating Self-supervised learning (SSL) models—Wav2Vec2, HuBERT, and WavLM—selected for their complementary strengths in phonetic, prosodic, and noisy speech modeling, with a hierarchical CNN-BiLSTM framework. Speech is segmented into breath-group chunks using Silero voice activity detection (Silero-VAD), enabling fine-grained temporal analysis while mitigating over-segmentation artifacts. SSL embeddings are fused via a learnable weighted mechanism, balancing acoustic and linguistic features, and enriched with chunk-level fluency markers (e.g., speech rate, pause durations, n-gram repetitions). The CNN-BiLSTM captures local and long-term dependencies across chunks. Evaluated on Avalinguo and Speechocean762, our approach improves F1-score by 2.8 and Pearson correlation by 6.2 points over single SSL baselines on Speechocean762, with gains of 4.2 F1-score and 4.0 Pearson points on Avalinguo, surpassing Pyannote.audio-based segmentation baselines. These findings highlight chunk-based multi-SSL fusion for robust fluency evaluation, though future work should explore generalization to dialects with irregular prosody.

Fichier principal
Vignette du fichier
PSW_EUSIPCO_2025.pdf (593.93 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05485978 , version 1 (30-01-2026)

Licence

Identifiants

Citer

Papa Séga Wade, Mihai Andries, Ioannis Kanellos, Thierry Moudenc. CBF-AFA: Chunk-Based Multi-SSL Fusion for Automatic Fluency Assessment. 33rd European Signal Processing Conference (EUSIPCO 2025), European Association for Signal Processing (EURASIP), Sep 2025, Isola delle Femmine – Palermo, Italy. ⟨10.23919/EUSIPCO63237.2025.11226329⟩. ⟨hal-05485978⟩
77 Consultations
108 Téléchargements

Altmetric

Partager

  • More