Color difference between views of a stereo pair is a challenging problem. Applications such as compression of stereo image demands the compensation of color differences which is typically done by methods called color mapping. Color mapping is based on feature correspondences. From these feature correspondences, color correspondences are generated which is ultimately used for the color mapping model. This paper focuses on detection of outliers in the feature correspondences. We propose novel iterative outlier removal method which exploits the neighborhood color information of the feature correspondences. From the analysis of our experimental results and comparing with existing methods we conclude by arguing that spatial color neighborhood information around the feature correspondences along with an iterative color mapping can detect outliers in general and can bring a robust color correction