mALBERT: Is a Compact Multilingual BERT Model Still Worth It? - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2024

mALBERT: Is a Compact Multilingual BERT Model Still Worth It?

Résumé

Within the current trend of Pretained Language Models (PLM), emerge more and more criticisms about the ethical and ecological impact of such models. In this article, considering these critical remarks, we propose to focus on smaller models, such as compact models like ALBERT, which are more ecologically virtuous than these PLM. However, PLMs enable huge breakthroughs in Natural Language Processing tasks, such as Spoken and Natural Language Understanding, classification, Question–Answering tasks. PLMs also have the advantage of being multilingual, and, as far as we know, a multilingual version of compact ALBERT models does not exist. Considering these facts, we propose the free release of the first version of a multilingual compact ALBERT model, pre-trained using Wikipedia data, which complies with the ethical aspect of such a language model. We also evaluate the model against classical multilingual PLMs in classical NLP tasks. Finally, this paper proposes a rare study on the subword tokenization impact on language performances.
Fichier principal
Vignette du fichier
mALBERT_final.pdf (231.91 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04520797 , version 1 (26-03-2024)

Licence

Domaine public

Identifiants

  • HAL Id : hal-04520797 , version 1

Citer

Christophe Servan, Sahar Ghannay, Sophie Rosset. mALBERT: Is a Compact Multilingual BERT Model Still Worth It?. The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, May 2024, Torino, Italy. ⟨hal-04520797⟩
3 Consultations
0 Téléchargements

Partager

Gmail Facebook X LinkedIn More