With the maturity of machine learning methods to provide satisfying Content-Based Image Retrieval systems (CBIR), research focus has recently turned back towards visual saliency analysis. The goal in these works is to extract even more efficient visual features than the existing ones. However, analyzing visual saliency is critically dependent on the task to be accomplished from the extracted visual features. A significant number of CBIR systems consider image retrieval as a binary classification problem: what is relevant for the user against what is irrelevant. In this paper, we focus on extracting relevant gaze features within the paradigm of visual preference in order to support the annotation by gaze for a CBIR system. We thus define a gaze acquisition protocol, design a benchmark from a subset of Pascal VOC database and present an in depth analysis of eye-tracking data for visual preference paradigm. Our paper provides new informations on relevant gaze features for image binary classification.