Superpixel decomposition methods are widely used in computer vision and image processing frameworks. By reducing the set of pixels to process, the computational burden can be drastically reduced. For most superpixel methods, a trade-off is computed between 1) color homogeneity, 2) adherence to the image contours and 3) shape regularity of the decomposition. In this paper, we propose an iterative framework that jointly enforces all these aspects and provides accurate and robust Superpixels with Contour Adherence using Linear Path (SCALP). The resulting superpixels adhere to the image contours while their regularity is enforced. During the decomposition process , we propose to compute the color distance along the linear path between the pixel and the corresponding superpixel barycenter. A contour prior is also considered on this path to prevent the crossing of image boundaries when associating a pixel to a superpixel. Finally, in order to improve the clustering accuracy and the robustness to noise, we integrate the pixel neighborhood information in the decomposition, while preserving the same computational complexity. SCALP is extensively evaluated on the standard Berkeley segmentation dataset, and the obtained results outperform the ones of state-of-the-art methods in terms of superpixel and contour detection metrics. The method is also extended to generate volume supervoxels, and evaluated on 3D MRI segmentation.