Visual learning with weak supervision is a promising re-search area, since it offers the possibility to build large image datasets at reasonable cost. In this paper, we address the prob-lem of weakly supervised object detection, where the goal is to predict the label of the image using object position as latent variable. We propose a new method that builds upon the La-tent Structural SVM (LSSVM) formalism. Specifically, we introduce an original coarse-to-fine approach that limits the evolution of the latent parameter subspace. This incremental strategy drives the learning towards better solutions, provid-ing a model with increased predictive accuracy. In addition, this leads to a significant speed up during learning and infer-ence compared to standard sliding window methods. Experi-ments carried out on Mammal dataset validate the good per-formances and fast training of the method compared to state-of-the-art works.