This paper presents a novel unrolled optimization method to reconstruct a dense light field from a focal stack containing only very few images captured with different focus. The proposed unrolled method first reconstructs Fourier Disparity Layers (FDL) from which all the light field viewpoints can then be computed. By recovering details in regions that are out-of-focus in all the captured images, the produced FDL model is also suitable for post-capture scene refocusing from a sparse focal stack. Solving the optimization problem in the FDL domain allows us to derive a closed-form expression of the data-fit term of the inverse problem. We show that the proposed framework outperforms state-of-the-art methods from focal stack measurements for both light field reconstruction and image refocusing.