Unsupervised Creation of Normalisation Dictionaries for Micro-Blogs in Arabic, French and English - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2018

Unsupervised Creation of Normalisation Dictionaries for Micro-Blogs in Arabic, French and English

Résumé

Text normalisation is a necessity to correct and make more sense of the micro-blogs messages, for information retrieval purposes. Unfortunately, tools and resources of text normalisation are rarely shared. In this paper, an approach is presented based on an unsupervised method for text normalisation using distributed representations of words, known also as "word embedding", applied on Arabic, French and English Languages. In addition, a tool will be supplied to create dictionaries for micro-blogs normalisation, in a form of pairs of misspelled word with its standard-form word, in the languages: Arabic, French and English. The tool will be available as open source including the resources: word embedding's models (with vocabulary size of million words for Arabic language model, million words for English language model and thousand words for French language model), and also three normalisation dictionaries of thousand pairs in Arabic language, thousand pairs in French language and thousand pairs in English language. The evaluation of the tool shows an average in N ormalisation success of % for English language, .% for Arabic Language and % for French Language. Also, the results of using an English language normalisation dictionary with a sentiment analysis tool for micro-blog's messages, show an increase in f-measure from. to ..
Fichier principal
Vignette du fichier
AmalHtait_CICLing2018.pdf (485.73 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01795348 , version 1 (18-05-2018)

Identifiants

  • HAL Id : hal-01795348 , version 1

Citer

Amal Htait, Sébastien Fournier, Patrice Bellot. Unsupervised Creation of Normalisation Dictionaries for Micro-Blogs in Arabic, French and English. 19th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing 2018), Mar 2018, Hanoi, Vietnam. ⟨hal-01795348⟩
135 Consultations
68 Téléchargements

Partager

Gmail Facebook X LinkedIn More