Investigating the Effect of Discourse Connectives on Transformer Surprisal: Language Models Understand Connectives; Even So They Are Surprised - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Investigating the Effect of Discourse Connectives on Transformer Surprisal: Language Models Understand Connectives; Even So They Are Surprised

Yan Cong
  • Fonction : Auteur
Emmanuele Chersoni
  • Fonction : Auteur
Yu-Yin Hsu
  • Fonction : Auteur
Philippe Blache

Résumé

As neural language models (NLMs) based on Transformers are becoming increasingly dominant in natural language processing, several studies have proposed analyzing the semantic and pragmatic abilities of such models. In our study, we aimed at investigating the effect of discourse connectives on NLMs with regard to Surprisal scores. We did this by focusing on the English stimuli of an experimental dataset, in which the expectations about an event in a discourse fragment could be reversed by a concessive or a contrastive connective. By comparing the Surprisal scores of several NLMs, we found that bigger NLMs show patterns similar to humans' behavioral data when a concessive connective is used, while connective-related effects tend to disappear with a contrastive one. We have additionally validated our findings with GPT-Neo using an extended dataset, and results mostly show a consistent pattern.
Fichier principal
Vignette du fichier
2023.blackboxnlp-1.17.pdf (1.03 Mo) Télécharger le fichier
Origine : Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-04471861 , version 1 (21-02-2024)

Identifiants

Citer

Yan Cong, Emmanuele Chersoni, Yu-Yin Hsu, Philippe Blache. Investigating the Effect of Discourse Connectives on Transformer Surprisal: Language Models Understand Connectives; Even So They Are Surprised. 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, Dec 2023, Singapore, Singapore. ⟨10.18653/v1/2023.blackboxnlp-1.17⟩. ⟨hal-04471861⟩
4 Consultations
2 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More