Analysis of nanoindentation experiments assumes that the indentation occurs on a flat surface. As a result, the accuracy of nanoindentation depends on reducing the surface roughness to a tolerable level. Within the context of statistical nanoindentation techniques suitable for heterogeneous materials, this study presents a criterion for roughness of cement paste surfaces for nanoindentation, and describes a method for obtaining the desired roughness. Through a systematic experimental study, we show the evolution of roughness and nanomechanical properties from indentation as a function of increased polishing. We conclude that the root-mean-squared (RMS) roughness of the sample, taken over a square area with edge dimensions of 200 times the average indentation depth of the dominating phase of the material, should be less than five times the average indentation depth of the dominating phase of the material.