On Spammer Detection in Crowdsourcing Pairwise Comparison Tasks: Case Study on Two Multimedia QoE Assessment Scenarios. - Archive ouverte HAL Access content directly
Conference Papers Year : 2021

On Spammer Detection in Crowdsourcing Pairwise Comparison Tasks: Case Study on Two Multimedia QoE Assessment Scenarios.

Abstract

The last decade has brought a surge in crowdsourcing platforms' popularity for the subjective quality evaluation of multimedia content. The lower need for intervention during the experiment and more expansive participant pools of crowdsourcing platforms encourage researchers to join this trend. However, the unreliability of the participant behaviors puts a barrier in the wide adoption of these platforms. Although many works exist to detect unreliable observers in rating experiments, there is still a lack of methodology for detecting unreliable observers in quality evaluation of multimedia content using pairwise comparison. In this work, we propose methods to identify irregular annotator behaviors in pairwise comparison paradigm. We compare the proposed methods' efficiency for two scenarios: quality evaluation of traditional 2D images and 3D interactive multimedia. We conducted two crowdsourcing experiments for two different Quality of Experience assessment tasks and inserted carefully designed synthetic spammer profiles to evaluate the proposed tools. Our results suggest that the detection of unreliable observers is highly task-dependent. The influence of the spammer behavior intensity and the proportion of spammers among the observers can be more severe on tasks with higher subjectivity. Based on these findings, we provide guidelines and recommendations towards developing spammer detection algorithms for subjective pairwise quality evaluation of multimedia content.
Fichier principal
Vignette du fichier
ICMEW_2021_Camera_Ready.pdf (3.17 Mo) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03236236 , version 1 (26-05-2021)

Identifiers

Cite

Ali Ak, Mona Abid, Matthieu Perreira da Silva, Patrick Le Callet. On Spammer Detection in Crowdsourcing Pairwise Comparison Tasks: Case Study on Two Multimedia QoE Assessment Scenarios.. ICME 2021 - First International Workshop on Quality of Experience in Interactive Multimedia, Jul 2021, Virtual, China. ⟨10.1109/ICMEW53276.2021.9455992⟩. ⟨hal-03236236⟩
90 View
138 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More