The objective of this paper is to investigate the problem of how to best combine and fuse color and depth measurements for incremental pose estimation or 3D tracking. Subsequently a framework will be proposed that allows to formulate the problem with a unique measurement vector and not to combine them in an ad-hoc manner. In particular, the full color and depth measurement will be defined as a 4-vector (by combining 3D Euclidean points + image intensities) and an optimal error for pose estimation will be derived from this. As will be shown, this will lead to designing an iterative closest point approach in 4-dimensional space. A kd-tree is used to find the closest point in 4-space, therefore simultaneously accounting for color and depth. Based on this unified framework a novel point-to-hyperplane approach will be introduced which has the advantages of classic point-to-plane ICP but in 4-space. By doing this it will be shown that there is no longer any need to provide or estimate a scale-factor between different measurement types. Consequently this allows to increase the convergence domain and speed up the alignment, whilst maintaining the robust and accurate properties. Results on both simulated and real environments will be provided along with benchmark comparisons.