We investigate - both experimentally and theoretically - the inelastic interaction between fast electrons and the electromagnetic field scattered by metallic apertures and nanostructures on dielectric membranes using photon induced near-field electron microscopy. The experiments - performed in a high brightness ultrafast transmission electron microscope - on gold apertures on silicon nitride membranes reveal strong modulations of the electron-light coupling strength. We demonstrates that this effect results from the combined action of the electric field scattered by the aperture edges and the reflection and transmission of the incident wave by the dielectric membrane. Moreover, when a nanostructure is added inside the metallic aperture, the new scattered field interferes with the previous contributions, thus imprinting the optical response of the nanostructure in additional modulations of the electron-light coupling strength. Using systematic electrodynamics simulations based on the Green dyadic method, we quantitatively analyze these different contributions to the electron-light coupling and propose further applications.