GlotLID: Language Identification for Low-Resource Languages - Archive ouverte HAL
Communication Dans Un Congrès Année : 2023

GlotLID: Language Identification for Low-Resource Languages

Résumé

Several recent papers have published good solutions for language identification (LID) for about 300 high-resource and medium-resource languages. However, there is no LID available that (i) covers a wide range of low-resource languages, (ii) is rigorously evaluated and reliable and (iii) efficient and easy to use. Here, we publish GlotLID-M, an LID model that satisfies the desiderata of wide coverage, reliability and efficiency. It identifies 1665 languages, a large increase in coverage compared to prior work. In our experiments, GlotLID-M outperforms four baselines (CLD3, FT176, OpenLID and NLLB) when balancing F1 and false positive rate (FPR). We analyze the unique challenges that low-resource LID poses: incorrect corpus metadata, leakage from high-resource languages, difficulty separating closely related languages, handling of macrolanguage vs varieties and in general noisy data. We hope that integrating GlotLID-M into dataset creation pipelines will improve quality and enhance accessibility of NLP technology for low-resource languages and cultures. GlotLID-M model, code, and list of data sources are available: https: //github.com/cisnlp/GlotLID.
Fichier principal
Vignette du fichier
2023.findings-emnlp.410.pdf (600.8 Ko) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte
Licence

Dates et versions

hal-04332442 , version 1 (08-12-2023)

Licence

Identifiants

  • HAL Id : hal-04332442 , version 1

Citer

Amir Hossein Kargaran, Ayyoob Imani, François Yvon, Hinrich Schütze. GlotLID: Language Identification for Low-Resource Languages. Findings of the Association for Computational Linguistics: EMNLP 2023, Association for Computational Linguistics, Dec 2023, Singapore, Singapore. pp.6155-6218. ⟨hal-04332442⟩
24 Consultations
167 Téléchargements

Partager

More