Moving Objects Segmentation (MOS) is critical and indispensable for secure intelligent vehicle operation in the dynamic environment. For the state estimation task which is based on the assumption of static surroundings, to identify and filter out the moving objects plays an important role in robust ego-motion estimation. In this paper, a LiDAR-Vision fusion approach is developed to segment moving objects in the scene, which utilizes the LiDAR-based semantic segmentation as a prior and vision-based geometric information for validation. The effectiveness of our approach to segment moving objects is highlighted by the comparison with the traditional robust kernel-based outlier rejection methods. Our approach is benchmarked with three city category sequences in the KITTI dataset, which outperforms the kernel-based methods and achieves the leading results of 77.9% average fitness and 7.65 cm RMSE respectively.