The impact of surface roughness on conductive heat transfer across nanoscale contacts is investigated by means of scanning thermal microscopy. Silicon surfaces with out-of-plane rms roughness of ~0, 0.5, 4, 7 and 11 nm are scanned both under air and vacuum conditions. Three types of resistive SThM probes spanning curvature radii over orders of magnitude are used. A correlation between thermal conductance and adhesion force is highlighted. In comparison with a flat surface, the contact thermal conductance can decrease as much as 90% for a microprobe and by about 50% for probes with curvature radius lower than 50 nm. The effects of multi-contact and ballistic heat conduction are discussed. Limits of contact techniques for thermal conductivity characterization are also discussed.