Amazon Mechanical Turk: Gold Mine or Coal Mine ? - Archive ouverte HAL
Article Dans Une Revue Computational Linguistics Année : 2011

Amazon Mechanical Turk: Gold Mine or Coal Mine ?

Résumé

Recently heard at a tutorial in our field: “It cost me less than one hundred bucks to annotate this using Amazon Mechanical Turk!” Assertions like this are increasingly common, but we believe they should not be stated so proudly; they ignore the ethical consequences of using MTurk (Amazon Mechanical Turk) as a source of labour. Manually annotating corpora or manually developing any other linguistic resource, such as a set of judgments about system outputs, represents such a high cost that many researchers are looking for alternative solutions to the standard approach. MTurk is becoming a popular one. However, as in any scientific endeavor involving humans, there is an unspoken ethical dimension involved in resource construction and system evaluation, and this is especially true of MTurk. We would like here to raise some questions about the use of MTurk. To do so, we will define precisely what MTurk is and what it is not, highlighting the issues raised by the system. We hope that this will point out opportunities for our community to deliberately value ethics above cost savings.
Fichier principal
Vignette du fichier
LastWords_AMT.pdf (72.97 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-00569450 , version 1 (25-02-2011)

Licence

Identifiants

Citer

Karen Fort, Gilles Adda, Kevin Bretonnel Cohen. Amazon Mechanical Turk: Gold Mine or Coal Mine ?. Computational Linguistics, 2011, 37 (2), pp.413-420. ⟨10.1162/COLI_a_00057⟩. ⟨hal-00569450⟩
534 Consultations
2359 Téléchargements

Altmetric

Partager

More