In this paper, we introduce the challenges, intrinsic and extrinsic, of color and depth sensors integration in the same matrix for a monolithic RBG-Z CMOS imager system. Due to the fact that the technology to conceive this type of circuit is still under development, the challenge that we address is the extrinsic one. It is a consequence of the heterogeneity of the matrix, where information is missing compared to what can be provided by separate RGB and Z systems. For that a first evaluation is done taking into account how the RGB-Z patterns could impact the demosaicing step. The evaluated pattern are in function of the different sizes between color and depth pixels. For the missing color reconstruction we have evaluated the state of the art algorithms, adapted to the missing information, and we propose an original adaptive algorithm using a new operator called semi-gradient (SG). To fill the lack of a mature technology for which real images are missing for this type of CMOS imager, a test environment was created and then used with three different databases, Kodak, McMaster, HDR+burst. The results show improvements on edges, corners, and narrow lines reconstruction, and a reduction of color and structural artefacts compared to the state-of-the-art reconstruction algorithms.