This paper proposes a novel two-stage approach for detecting moving objects with a non-stationary monocular camera mounted on a vehicle. We formulate an innovative method called semantic-guided random sample consensus (Semantic-Guided RANSAC) to detect moving objects by semantic-geometric information fusion and integration. Firstly, semantic constraints from deep learning architecture (YOLO v4) are applied to predict the objects' location in the image frame. The fundamental matrix is then estimated robustly from two views through the sparse optical flow tracking with the help of semantic prior. Semantic-guided RANSAC is used to reject instancelevel outliers which are actually moving objects based on the epipolar geometry and flow vector bound constraints. Experimental results on KITTI dataset reflect the effectiveness of our approach to identify moving objects in complex urban traffic scenes with the average precision above 0.82 for 4 sequences in the City category.