In this paper, we introduce a novel color pixel reconstruction algorithm for a RGBZ CMOS imager. A RGBZ imager is a Vision System on Chip (VSoC) sensor that captures simultaneously color and depth information with a hybrid pixel matrix. The imager pattern is based on a Bayer filter array where the Z pixel covers an equivalent area of a 2x2-color-pixel size, and one quarter of the total color information is missing. We propose an original adaptive algorithm to reconstruct the color information using a new operator called semi-gradient. The results show improvements on edges, corners, and narrow lines reconstruction, and a reduction of color and structural artefacts compared to classical reconstruction algorithms.