The detection of moving objects in the video stream of a moving camera is a complex task. Static objects appear moving in the video stream as moving objects. Thus, it is difficult to identify motions that belong to moving objects because they are hidden by those of static objects. To detect moving objects we propose a novel geometric constraint based on 2D and 3D information. A sparse reconstruction of the visible part of the scene is performed in order to detect motions in the 3D space where the scene perception is not deformed by the camera motion. A first labeling estimation is performed in the 3D space and then apparent motions in the video stream of the moving camera are used to validate the estimation. Labels are computed from confidence values which are updated at each frame according to the geometric constraint. Our method can detect several moving objects in complex scenes with high parallax.