Communication Dans Un Congrès Année : 2025

An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training

Résumé

This paper investigates discrete unit representations in Speech Language Models (SLMs), focusing on optimizing speech modeling during continual pre-training. In this paper, we systematically examine how model architecture, data representation, and training robustness influence the pre-training stage in which we adapt existing pre-trained language models to the speech modality. Our experiments highlight the role of speech encoders and clustering granularity across different model scales, showing how optimal discretization strategies vary with model capacity. By examining cluster distribution and phonemic alignments, we investigate the effective use of discrete vocabulary, uncovering both linguistic and paralinguistic patterns. Additionally, we explore the impact of clustering data selection on model robustness, highlighting the importance of domain matching between discretization training and target applications.

Fichier principal
Vignette du fichier
_TSD_2025__An_Empirical_Analysis_of_Discrete_Unit_Representations_in_Speech_Language_Modeling_Pre_training-8.pdf (1.05 Mo) Télécharger le fichier

Dates et versions

hal-05225350 , version 1 (02-09-2025)

Licence

Identifiants

Citer

Yanis Labrak, Richard Dufour, Mickaël Rouvier. An Empirical Analysis of Discrete Unit Representations in Speech Language Modeling Pre-training. International Conference on Text, Speech, and Dialogue, Kamil Ekštein; Miloslav Konopík; Ondřej Pražák; František Pártl, Aug 2025, Erlangen, Germany. pp.13-24, ⟨10.1007/978-3-032-02548-7_2⟩. ⟨hal-05225350⟩
391 Consultations
186 Téléchargements

Altmetric

Partager

  • More