Correlation is widely used to test the hypothesis of the relationship between two variables. In this paper we chose to focus the discussion on query difficulty prediction for which correlation is often used to measure the accuracy of predictors. Here, the correlation is calculated between the actual system effectiveness and the predicted one. Although fairly simple to calculate, the Pearson correlation coefficient can be difficult to interpret and use correctly, especially because of its sensitivity to outliers. This paper illustrates the problem and opens discussion pathways.