AdminSet and AdminBERT: a Dataset and a Pre-trained Language Model to Explore the Unstructured Maze of French Administrative Documents - Archive ouverte HAL
Communication Dans Un Congrès Année : 2025

AdminSet and AdminBERT: a Dataset and a Pre-trained Language Model to Explore the Unstructured Maze of French Administrative Documents

Résumé

In recent years, Pre-trained Language Models (PLMs) have been widely used to analyze various documents, playing a crucial role in Natural Language Processing (NLP). However, administrative texts have rarely been used in information extraction tasks, even though this resource is available as open data in many countries. Most of these texts contain many specific domain terms. Moreover, especially in France, they are unstructured because many administrations produce them without a standardized framework. Due to this fact, current language models do not process these documents correctly. In this paper, we propose AdminBERT, the first French pre-trained language model for the administrative domain. As interesting information in such texts correspond to named entities and the relations between them, we compare this PLM to general domain language models, fine-tuned on the Named Entity Recognition (NER) task applied to administrative texts, as well as to a Large Language Model (LLM) and to a language model with an architecture different to the BERT one. We show that taking advantage of a PLM for French administrative data increases the performance in the administrative and general domains, on these texts. We also release AdminBERT as well as Admin-Set, the pre-training corpus of administrative texts in French and the subset AdminSet-NER, the first NER dataset consisting exclusively of administrative texts in French.

Fichier sous embargo
Fichier sous embargo
0 0 18
Année Mois Jours
Avant la publication
samedi 25 janvier 2025
Fichier sous embargo
samedi 25 janvier 2025
Connectez-vous pour demander l'accès au fichier

Dates et versions

hal-04855066 , version 1 (24-12-2024)

Licence

Identifiants

  • HAL Id : hal-04855066 , version 1

Citer

Thomas Sebbag, Solen Quiniou, Nicolas Stucky, Emmanuel Morin. AdminSet and AdminBERT: a Dataset and a Pre-trained Language Model to Explore the Unstructured Maze of French Administrative Documents. Conference on Computational Linguistics (COLING 2025), Jan 2025, Abu Dhabi, United Arab Emirates. ⟨hal-04855066⟩
0 Consultations
0 Téléchargements

Partager

More