Mean shift, a powerful color clustering approach successfully applied to image segmentation, has two main properties that are relevant for use in document image segmentation. These properties include: the autonomous definition of both color clusters' centers and numbers and the good tolerance to noisy data sets. Hence, mean shift could robustly process degraded background document images and improve their legibility. Nevertheless, this paper proves that coupling this approach and anisotropic diffusion within a joint iterative framework has more interesting results. For instance, this framework generates segmented images with more reduced artefacts on edges and background than those obtained after applying each method alone. This improvement is explained by the mutual interaction of global and local information, respectively introduced by the mean shift and anisotropic diffusion, and by the nature of this latter, smoothing while preserving continuities across edges. Some experiments, done on real ancient document images, illustrate these ideas and indicate that our proposed framework provides an efficient tool for document image segmentation and restoration.