The Self-Contained Negation Test Set - Archive ouverte HAL
Communication Dans Un Congrès Année : 2023

The Self-Contained Negation Test Set

Résumé

Several methodologies have recently been proposed to evaluate the ability of Pretrained Language Models (PLMs) to interpret negation. In this article, we build on Gubelmann and Handschuh (2022), which studies the modification of PLMs’ predictions as a function of the polarity of inputs, in English. Crucially, this test uses “self-contained” inputs ending with a masked position: depending on the polarity of a verb in the input, a particular token is either semantically ruled out or allowed at the masked position. By replicating Gubelmann and Handschuh (2022) experiments, we have uncovered flaws that weaken the conclusions that can be drawn from this test. We thus propose an improved version, the Self-Contained Neg Test, which is more controlled, more systematic, and entirely based on examples forming minimal pairs varying only in the presence or absence of verbal negation in English. When applying our test to the roberta and bert base and large models, we show that only roberta-large shows trends that match the expectations, while bert-base is mostly insensitive to negation. For all the tested models though, in a significant number of test instances the top-1 prediction remains the token that is semantically forbidden by the context, which shows how much room for improvement remains for a proper treatment of the negation phenomenon.
Fichier principal
Vignette du fichier
2023.blackboxnlp-1.16.pdf (253.76 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04671338 , version 1 (14-08-2024)

Licence

Identifiants

Citer

David Kletz, Pascal Amsili, Marie Candito. The Self-Contained Negation Test Set. BlackboxNLP 2024, BlackboxNLP, Dec 2023, Singapour, Singapore. pp.212-221. ⟨hal-04671338⟩
16 Consultations
11 Téléchargements

Altmetric

Partager

More