How Dropping Subjects Who Failed Manipulation Checks Can Bias Your Results: An Illustrative Case - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Journal of Experimental Political Science Année : 2022

How Dropping Subjects Who Failed Manipulation Checks Can Bias Your Results: An Illustrative Case

Simon Varaine

Résumé

Manipulations checks are postexperimental measures widely used to verify that subjects understood the treatment. Some researchers drop subjects who failed manipulation checks in order to limit the analyses to attentive subjects. This short report offers a novel illustration on how this practice may bias experimental results: in the present case, through confirming a hypothesis that is likely false. In a survey experiment, subjects were primed with a fictional news story depicting an economic decline versus prosperity. Subjects were then asked whether the news story depicted an economic decline or prosperity. Results indicate that responses to this manipulation check captured subjects’ preexisting beliefs about the economic situation. As a consequence, dropping subjects who failed the manipulation check mixes the effects of preexisting and induced beliefs, increasing the risk of false positive findings. Researchers should avoid dropping subjects based on posttreatment measures and rely on pretreatment measures of attentiveness.
Fichier principal
Vignette du fichier
How_dropping_subjects_who_failed_manipulation_checks.pdf (390.95 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04046167 , version 1 (01-04-2023)

Licence

Paternité

Identifiants

Citer

Simon Varaine. How Dropping Subjects Who Failed Manipulation Checks Can Bias Your Results: An Illustrative Case. Journal of Experimental Political Science, 2022, pp.1-7. ⟨10.1017/XPS.2022.28⟩. ⟨hal-04046167⟩

Collections

UGA CNRS PACTE
28 Consultations
318 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More