The complexity of measuring reliability in learning tasks: An illustration using the Alternating Serial Reaction Time Task - Archive ouverte HAL
Article Dans Une Revue Behavior Research Methods Année : 2024

The complexity of measuring reliability in learning tasks: An illustration using the Alternating Serial Reaction Time Task

Résumé

Despite the fact that reliability estimation is crucial for robust inference, it is underutilized in neuroscience and cognitive psychology. Appreciating reliability can help researchers increase statistical power, effect sizes, and reproducibility, decrease the impact of measurement error, and inform methodological choices. However, accurately calculating reliability for many experimental learning tasks is challenging. In this study, we highlight a number of these issues, and estimate multiple metrics of internal consistency and split-half reliability of a widely used learning task on a large sample of 180 subjects. We show how pre-processing choices, task length, and sample size can affect reliability and its estimation. Our results show that the Alternating Serial Reaction Time Task has respectable reliability, especially when learning scores are calculated based on reaction times and two-stage averaging. We also show that a task length of 25 blocks can be sufficient to meet the usual thresholds for minimally acceptable reliability. We further illustrate how relying on a single point estimate of reliability can be misleading, and the calculation of multiple metrics, along with their uncertainties, can lead to a more complete characterization of the psychometric properties of tasks.
Fichier principal
Vignette du fichier
13428_2022_Article_2038.pdf (2.75 Mo) Télécharger le fichier
Origine Publication financée par une institution

Dates et versions

hal-04626798 , version 1 (30-08-2024)

Licence

Identifiants

Citer

Bence C. Farkas, Attila Krajcsi, Karolina Janacsek, Dezso Nemeth. The complexity of measuring reliability in learning tasks: An illustration using the Alternating Serial Reaction Time Task. Behavior Research Methods, 2024, 56 (1), pp.301-317. ⟨10.3758/s13428-022-02038-5⟩. ⟨hal-04626798⟩
45 Consultations
3 Téléchargements

Altmetric

Partager

More