In this article, we propose a novel approach to detect moving objects in H.264 compressed bitstreams. More precisely, we describe a multi-modal background subtraction technique that uses the size of macroblocks in order to label them as belonging to the background of the observed scene or not. Here, we integrate an adaptive Gaussian mixture-based scheme to model the background. We evaluate our contribution using the PETS video dataset and a realist synthetic video sequence rendered by a 3-D urban environment simulator. We compare two different background models, and we show that the Gaussian mixture-based is the best and outperforms other techniques that use macro bloc sizes.