We propose the deep Gauss–Newton (DGN) algorithm. The DGN allows one to take into account the knowledge of the forward model in a deep neural network by unrolling a Gauss–Newton optimization method. No regularization or step size needs to be chosen; they are learned through convolutional neural networks. The proposed algorithm does not require an initial reconstruction and is able to retrieve simultaneously the phase and absorption from a single-distance diffraction pattern. The DGN method was applied to both simulated and experimental data and permitted large improvements of the reconstruction error and of the resolution compared with a state-of-the-art iterative method and another neural-network-based reconstruction algorithm.