In this paper we design a smoothing filter for texture+depth images based on anisotropic diffusion. Our proposed filter enables to generate a scale space on the texture image guided by depth information , and is linear and numerically stable. We show experimentally that using scene geometry preserves the internal structure of 3D surfaces (e.g., it avoids smoothing across object boundaries). As a consequence, the result of smoothing is more independent to changes in the camera position. To illustrate the practical utility of a scale space with such properties, we integrate our filter into the SIFT key-point detector, getting a substantial improvement of the repeatability of detected keypoints under significant viewpoint position changes.