Unsupervised Creation of Normalisation Dictionaries for Micro-Blogs in Arabic, French and English
Résumé
Text normalisation is a necessity to correct and make more sense of the micro-blogs messages, for information retrieval purposes. Unfortunately, tools and resources of text normalisation are rarely shared. In this paper, an approach is presented based on an unsupervised method for text normalisation using distributed representations of words, known also as "word embedding", applied on Arabic, French and English Languages. In addition, a tool will be supplied to create dictionaries for micro-blogs normalisation, in a form of pairs of misspelled word with its standard-form word, in the languages: Arabic, French and English. The tool will be available as open source including the resources: word embedding's models (with vocabulary size of million words for Arabic language model, million words for English language model and thousand words for French language model), and also three normalisation dictionaries of thousand pairs in Arabic language, thousand pairs in French language and thousand pairs in English language. The evaluation of the tool shows an average in N ormalisation success of % for English language, .% for Arabic Language and % for French Language. Also, the results of using an English language normalisation dictionary with a sentiment analysis tool for micro-blog's messages, show an increase in f-measure from. to ..
Origine : Fichiers produits par l'(les) auteur(s)
Loading...