ALIBERT: A PRETRAINED LANGUAGE MODEL FOR FRENCH BIOMEDICAL TEXT A PREPRINT - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail (Preprint/Prepublication) Année : 2022

ALIBERT: A PRETRAINED LANGUAGE MODEL FOR FRENCH BIOMEDICAL TEXT A PREPRINT

Aman Berhe
  • Fonction : Auteur
  • PersonId : 1209387
Guillaume Draznieks
  • Fonction : Auteur
  • PersonId : 1209384
Vincent Martenot
  • Fonction : Auteur
  • PersonId : 1131265
Valentin Masdeu
  • Fonction : Auteur
  • PersonId : 1131266
Lucas Davy
  • Fonction : Auteur
  • PersonId : 1209385
Jean-Daniel Zucker

Résumé

Over the past few years, domain specific pretrained language models have been investigated and have shown remarkable achievements in different downstream tasks, especially in biomedical domain. These achievements stem on the well known BERT architecture which uses an attention based selfsupervision for context learning of textual documents. However, these domain specific biomedical pretrained language models mainly use English corpora. Therefore, non-English, domain-specific pretrained models remain quite rare, both of these requirements being hard to achieve. In this work, we proposed AliBERT, a biomedical pretrained language model for French and investigated different learning strategies. AliBERT is trained using regularized Unigram based tokenizer trained for this purpose. AliBERT has achieved state of the art F1 and accuracy scores in different downstream biomedical tasks. Our pretrained model manages to outperform some French non domain-specific models such as CamemBERT and FlauBERT on diverse downstream tasks, with less pretraining and training time and with much less corpora.
Fichier principal
Vignette du fichier
AliBERT_For_Submission.pdf (472.59 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03911564 , version 1 (23-12-2022)
hal-03911564 , version 2 (03-02-2023)

Identifiants

  • HAL Id : hal-03911564 , version 1

Citer

Aman Berhe, Guillaume Draznieks, Vincent Martenot, Valentin Masdeu, Lucas Davy, et al.. ALIBERT: A PRETRAINED LANGUAGE MODEL FOR FRENCH BIOMEDICAL TEXT A PREPRINT. 2022. ⟨hal-03911564v1⟩
539 Consultations
407 Téléchargements

Partager

Gmail Mastodon Facebook X LinkedIn More