This paper introduces a novel color tracking model for image registration that exploits directly the color information provided by standard color cameras. Furthermore, unlike previous approaches, the color tracking model is designed to handle both global and local illumination changes within a robust framework that also rejects outliers such as occluding objects, shadows, etc. In order to demonstrate the proposed approach a planar template tracking algorithm is used, however, the approach is also valid for a general class of direct tracking algorithms. In particular, the objective function is defined to be minimized directly in the CFA (color filter array) space instead of the common RGB space. It will be shown that this not only takes advantage of the discernibility of color measurements but also drastically improves the efficiency of the vision processing pipeline and ultimately improves real-time performance. A robust global illumination model is then combined with a robust M-estimation technique that is shown to handle outliers as well as global and local illumination changes. Results from synthetic and real sequences are presented to demonstrate the proposed concept.