The Hartman wavefront sensor can be used for X-ray phase imaging with high angular resolution. The Hartmann sensor is able to retrieve both the phase and absorption from a single acquisition. The system calculates the shift in a series of apertures imaged with a detector with respect to their reference positions. In this article, the impact of the reference image on the final image quality is investigated using a laboratory setup. Deflection and absorption images of the same sample are compared using reference images acquired in air and in water. It can be easily coupled with tomographic setups to obtain 3D images of both phase and absorption. Tomographic images of a test sample are shown, where deflection images revealed details that were invisible in absorption. The findings reported in this paper can be used for the improvement of image reconstruction and for expanding the applications of X-ray phase imaging towards materials characterization and medical imaging.