In this paper, we propose a new method for taking into ac-count the spatial information in image categorization. More specifically, we remove the loss of spatial information in Bag of Words related methods by computing the image signature over specific regions selected by object detectors. We propose to select the detectors using Multiple Kernel Learning tech-niques. We carry out experiments on the well known VOC 2007 dataset, and show our semantic pooling obtains promis-ing results.