In this paper, we propose an improvement of image retrieval for building images using the Bag of Words (BoW) model. The principle consists of pre-processing the interest points detected on the images in order to classify them into two classes, corresponding to building and no-building key points. In this way, the data involved for comparisons is reduced to only the relevant one and only the features describing the buildings are taken into account. The experimental results, carried out on the Paris6k data set shows significant improvement in terms of retrieval performances