We try to overcome the imbalanced data set problem in image annotation by choosing a convenient loss function for learning the classifier. Instead of training a standard SVM, we use a Ranking SVM in which the chosen loss function is helpful in the case of imbalanced data. We compare the Ranking SVM to a classical SVM with different visual features. We observe that Ranking SVM always improves the prediction quality, and can perform up to 23% better than the classical SVM.