Reducing the Cost of Aggregation in Crowdsourcing - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Transactions on Large-Scale Data- and Knowledge-Centered Systems Année : 2021

Reducing the Cost of Aggregation in Crowdsourcing

Résumé

Crowdsourcing is a way to solve problems that need human contribution. Crowdsourcing platforms distribute replicated tasks to workers, pay them for their contribution, and aggregate answers to produce a reliable conclusion. A fundamental problem is to infer a consensual answer from the set of returned results. Another problem is to obtain this answer at a reasonable cost: unlimited budget allows hiring experts or large pools of workers for each task but a limited budget forces to use resources at best. Last, crowdsourcing platforms have to detect and ban malevolent users (also known as "spammers") to achieve good accuracy of their answers. This paper considers crowdsourcing of simple Boolean tasks. We first define a probabilistic inference technique, that considers difficulty of tasks and expertise of workers when aggregating answers. We then propose CrowdInc, a greedy algorithm that reduces the cost needed to reach a consensual answer. CrowdInc distributes resources dynamically to tasks according to their difficulty. The algorithm solves batches of simple tasks in rounds that estimate workers expertize, tasks difficulty, and synthesizes a plausible aggregated conclusion and a confidence score using Expectation Maximization. The synthesized values are used to decide whether more workers should be hired to increase confidence in synthesized answers. We show on several benchmarks that CrowdInc achieves good accuracy, reduces costs and we compare its performance to existing solutions. We then use the estimation of CrowdInc to detect spammers and study the impact of spammers on costs and accuracy.
Fichier principal
Vignette du fichier
HelouetTLDKS21.pdf (3.82 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03482460 , version 1 (15-12-2021)

Identifiants

  • HAL Id : hal-03482460 , version 1

Citer

Rituraj Singh, Loïc Hélouët, Zoltan Miklos. Reducing the Cost of Aggregation in Crowdsourcing. Transactions on Large-Scale Data- and Knowledge-Centered Systems, In press, pp.1-38. ⟨hal-03482460⟩
36 Consultations
43 Téléchargements

Partager

Gmail Facebook X LinkedIn More