Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT

Résumé

Both humans and neural language models are able to perform subject-verb number agreement (SVA). In principle, semantics shouldn't interfere with this task, which only requires syntactic knowledge. In this work we test whether meaning interferes with this type of agreement in English in syntactic structures of various complexities. To do so, we generate both semantically well-formed and nonsensical items. We compare the performance of BERT-base to that of humans, obtained with a psycholinguistic online crowdsourcing experiment. We find that BERT and humans are both sensitive to our semantic manipulation: They fail more often when presented with nonsensical items, especially when their syntactic structure features an attractor (a noun phrase between the subject and the verb that has not the same number as the subject). We also find that the effect of meaningfulness on SVA errors is stronger for BERT than for humans, showing higher lexical sensitivity of the former on this task.
Fichier principal
Vignette du fichier
2022.coling-lasri_seminck_lenci_poibeau.pdf (279.71 Ko) Télécharger le fichier
Origine : Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-03812929 , version 1 (13-10-2022)

Identifiants

  • HAL Id : hal-03812929 , version 1

Citer

Karim Lasri, Olga Seminck, Alessandro Lenci, Thierry Poibeau. Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT. The 29th International Conference on Computational Linguistic (COLING 2022), Oct 2022, Gyeongju, South Korea. ⟨hal-03812929⟩
16 Consultations
33 Téléchargements

Partager

Gmail Facebook X LinkedIn More