Communication Dans Un Congrès Année : 2025

EuroBERT: Scaling Multilingual Encoders for European Languages

Duarte M. Alves
  • Fonction : Auteur
  • PersonId : 1574634
André F T Martins
  • Fonction : Auteur
  • PersonId : 1074793
Etienne Malaboeuf
Gabriel Hautreux
  • Fonction : Auteur
João Alves
  • Fonction : Auteur
Manuel Faysse
Ricardo Rei
  • Fonction : Auteur

Résumé

Equal contribution, † Ordered alphabetically by the first name, ⋆ Senior advisor General-purpose multilingual vector representations, used in retrieval, regression, and classification, are traditionally obtained from bidirectional encoder models. Despite their wide applicability, encoders have been recently overshadowed by advances in generative decoder-only models. However, many innovations driving this progress are not inherently tied to decoders. In this paper, we revisit the development of multilingual encoders through the lens of these advances, and introduce EuroBERT, a family of multilingual encoders covering European and widely spoken global languages. Our models outperform existing alternatives across a diverse range of tasks, spanning multilingual capabilities, mathematics, and coding, and natively support sequences of up to 8,192 tokens. We also examine the design decisions behind EuroBERT, offering insights into our dataset composition and training pipeline. We publicly release the EuroBERT models, including intermediate training checkpoints, together with our training framework.

Fichier principal
Vignette du fichier
2503.05500v2.pdf (1.96 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05226285 , version 1 (27-08-2025)

Licence

Identifiants

Citer

Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, André F T Martins, Ayoub Hammal, et al.. EuroBERT: Scaling Multilingual Encoders for European Languages. COLM 2025 - Second Conference on Language Modeling, Oct 2025, Montreal, Canada. pp.1-28. ⟨hal-05226285⟩
685 Consultations
1189 Téléchargements

Altmetric

Partager

  • More