Recovering quality scores in noisy pairwise subjective experiments using Negative log-likelihood - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

Recovering quality scores in noisy pairwise subjective experiments using Negative log-likelihood

Résumé

To gather larger datasets to train data-angry deep learning quality assessment models, crowdsourcing has become essential to recruit participants. These participants are asked their opinion by directly rating stimuli, e.g., using single or double stimulus methodologies, or indirectly by ranking stimuli or comparing distances as in the Maximum Likelihood Difference Scaling method. In crowdsourcing, participants' behaviors and environmental distractions are not controlled. So, the researcher must pay attention to the answers' reliability. Cleaning methods exist for direct annotation subjective methodologies. However, solutions for indirect annotation methods are limited. In this work, we propose a method based on the negative log-likelihood to detect spammers among participants from their answers. To demonstrate its use, we applied it in a quadruplet preference-based scenario. The proposed method requires low computation and can be integrated into active-sampling strategies, where annotations available per comparison are small. We demonstrate that our method is robust to various spammer behaviors and accurate by removing only spammers. It helps reduce the gap between data collected in in-lab conditions (i.e., no spammer) and through crowdsourcing: our method reduces estimated uncertainties around data-points by 50%, and RMSE between estimations from an in-lab experiment and the same experiment performed in crowdsourcing by 1.8.
Fichier principal
Vignette du fichier
ICIP2023_spammers_preference (1).pdf (577.31 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04132007 , version 1 (17-06-2023)

Identifiants

Citer

Andréas Pastor, Lukáš Krasula, Xiaoqing Zhu, Zhi Li, Patrick Le Callet. Recovering quality scores in noisy pairwise subjective experiments using Negative log-likelihood. 2023 IEEE International Conference on Image Processing (ICIP), Oct 2023, Kuala Lumpur, France. pp.2635-2639, ⟨10.1109/ICIP49359.2023.10222071⟩. ⟨hal-04132007⟩
67 Consultations
55 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More