Abstract In electron microscopy, detailed insights into nanoscale optical properties of materials are gained by spontaneous inelastic scattering leading to electron-energy loss and cathodoluminescence. Stimulated scattering in the presence of external sample excitation allows for mode- and polarization-selective photon-induced near-field electron microscopy (PINEM). This process imprints a spatial phase profile inherited from the optical fields onto the wave function of the probing electrons. Here, we introduce Lorentz-PINEM for the full-field, non-invasive imaging of complex optical near fields at high spatial resolution. We use energy-filtered defocus phase-contrast imaging and iterative phase retrieval to reconstruct the phase distribution of interfering surface-bound modes on a plasmonic nanotip. Our approach is universally applicable to retrieve the spatially varying phase of nanoscale fields and topological modes.