Making Sentence Embeddings Robust to User-Generated Content - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2024

Making Sentence Embeddings Robust to User-Generated Content

Résumé

NLP models have been known to perform poorly on user-generated content (UGC), mainly because it presents a lot of lexical variations and deviates from the standard texts on which most of these models were trained. In this work, we focus on the robustness of LASER, a sentence embedding model, to UGC data. We evaluate this robustness by LASER's ability to represent non-standard sentences and their standard counterparts close to each other in the embedding space. Inspired by previous works extending LASER to other languages and modalities, we propose RoLASER, a robust English encoder trained using a teacher-student approach to reduce the distances between the representations of standard and UGC sentences. We show that with training only on standard and synthetic UGC-like data, RoLASER significantly improves LASER's robustness to both natural and artificial UGC data by achieving up to 2x and 11x better scores. We also perform a fine-grained analysis on artificial UGC data and find that our model greatly outperforms LASER on its most challenging UGC phenomena such as keyboard typos and social media abbreviations. Evaluation on downstream tasks shows that RoLASER performs comparably to or better than LASER on standard data, while consistently outperforming it on UGC data.
Fichier principal
Vignette du fichier
Lydia_Nishimwe_LREC_COLING_2024.pdf (848.8 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Licence : CC BY - Paternité

Dates et versions

hal-04520909 , version 1 (25-03-2024)

Licence

Paternité

Identifiants

  • HAL Id : hal-04520909 , version 1

Citer

Lydia Nishimwe, Benoît Sagot, Rachel Bawden. Making Sentence Embeddings Robust to User-Generated Content. 2024. ⟨hal-04520909⟩
3 Consultations
2 Téléchargements

Partager

Gmail Facebook X LinkedIn More