Graph Neural Networks for Adapting Off-the-shelf General Domain Language Models to Low-Resource Specialised Domains - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Graph Neural Networks for Adapting Off-the-shelf General Domain Language Models to Low-Resource Specialised Domains

Résumé

Language models encode linguistic proprieties and are used as input for more specific models. Using their word representations as-is for specialised and low-resource domains might be less efficient. Methods of adapting them exist, but these models often overlook global information about how words, terms, and concepts relate to each other in a corpus due to their strong reliance on attention. We consider that global information can influence the results of the downstream tasks, and combination with contextual information is performed using graph convolution networks or GCN built on vocabulary graphs. By outperforming baselines, we show that this architecture is profitable for domain-specific tasks.

Dates et versions

hal-04517190 , version 1 (22-03-2024)

Identifiants

Citer

Mérième Bouhandi, Emmanuel Morin, Thierry Hamon. Graph Neural Networks for Adapting Off-the-shelf General Domain Language Models to Low-Resource Specialised Domains. 2nd Workshop on Deep Learning on Graphs for Natural Language Processing (DLG4NLP 2022), ACL, Jul 2022, Seattle, Washington, United States. pp.36-42, ⟨10.18653/v1/2022.dlg4nlp-1.5⟩. ⟨hal-04517190⟩
2 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More