Vehicles makers are now extremely concerned by the perceived quality of their products. For the design of many parts of vehicles, sensory profiling techniques are used, which are traditionally carried out by experts. These tests incur a significant financial effort and are time consuming. We propose in this paper an evaluation method with naive subjects, based on paired comparisons. Two studies have been carried out on diesel motor sounds: firstly, the panel of experts of a car maker made a conventional sensory profile and an evaluation by paired comparison. Secondly, 30 naive subjects also completed two tests (ratings and paired comparisons). For the experts, we noticed a very good agreement between the sensory profile and the paired comparisons. For the naives, the paired comparisons gave a better agreement between the subjects and were more discriminating than the ratings. The results of these two tests with the naives were then compared with the conventional sensory profile of the experts using Generalized Procrustes Analysis. Results show that the consensus is better with the paired comparison test. As a result, the evaluation method proposed with naives could be an interesting method for the perceptual evaluation of products.