We report on the implementation of ultracold atoms as a source in a state of the art atom gravimeter. We perform gravity measurements with 10 nm/s 2 statistical uncertainties in a so-far unexplored temperature range for such a high accuracy sensor, down to 50 nK. This allows for an improved characterization of the most limiting systematic effect, related to wavefront aberrations of light beam splitters. A thorough model of the impact of this effect onto the measurement is developed and a method is proposed to correct for this bias based on the extrapolation of the measurements down to zero temperature. Finally, an uncertainty of 13 nm/s 2 is obtained in the evaluation of this systematic effect, which can be improved further by performing measurements at even lower temperatures. Our results clearly demonstrate the benefit brought by ultracold atoms to the metrological study of free falling atom interferometers. By tackling their main limitation, our method allows reaching record-breaking accuracies for inertial sensors based on atom interferometry.