Probing structural constraints of negation in Pretrained Language Models - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Probing structural constraints of negation in Pretrained Language Models

Résumé

Contradictory results about the encoding of the semantic impact of negation in pretrained language models (PLMs). have been drawn recently (e.g. Kassner and Schütze (2020); Gubelmann and Handschuh (2022)). In this paper we focus rather on the way PLMs encode negation and its formal impact, through the phenomenon of the Negative Polarity Item (NPI) licensing in English. More precisely, we use probes to identify which contextual representations best encode 1) the presence of negation in a sentence, and 2) the polarity of a neighboring masked polarity item. We find that contextual representations of tokens inside the negation scope do allow for (i) a better prediction of the presence of not compared to those outside the scope and (ii) a better prediction of the right polarity of a masked polarity item licensed by not, although the magnitude of the difference varies from PLM to PLM. Importantly, in both cases the trend holds even when controlling for distance to not. This tends to indicate that the embeddings of these models do reflect the notion of negation scope, and do encode the impact of negation on NPI licensing. Yet, further control experiments reveal that the presence of other lexical items is also better captured when using the contextual representation of a token within the same syntactic clause than outside from it, suggesting that PLMs simply capture the more general notion of syntactic clause.

Mots clés

Fichier principal
Vignette du fichier
CR_Probing_structural_constraints_of_negation_in_PLMs_NoDaLiDA_long.pdf (453.91 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04641828 , version 1 (01-08-2024)

Licence

Identifiants

Citer

David Kletz, Marie Candito, Pascal Amsili. Probing structural constraints of negation in Pretrained Language Models. 24th Nordic Conference on Computational Linguistics (NoDaLiDa), Northern European Association for Language Technology (NEALT), May 2023, Tórshavn, Faroe Islands. pp.541-554. ⟨hal-04641828⟩
6 Consultations
14 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More