Observationally mapping the relation between galaxies and the intergalactic medium (IGM) is of key interest for studies of cosmic reionization. Diffuse hydrogen gas has typically been observed in HI Lyman-$\alpha$ (Ly$\alpha$) absorption in the spectra of bright background quasars. However, it is important to extend these measurements to background galaxies as quasars become increasingly rare at high redshift and rarely probe closely separated sight-lines. Here we use deep integral field spectroscopy in the MUSE eXtremely Deep Field to demonstrate the measurement of the Ly$\alpha$ transmission at $z\approx4$ in absorption to a background galaxy at $z=4.77$. The HI transmission is consistent with independent quasar sight-lines at similar redshifts. Exploiting the high number of spectroscopic redshifts of faint galaxies (500 between $z=4.0-4.7$ within a radius of 8 arcmin) that are tracers of the density field, we show that Ly$\alpha$ transmission is inversely correlated with galaxy density, i.e. transparent regions in the Ly$\alpha$ forest mark under-dense regions at $z\approx4$. Due to large-scale clustering, galaxies are surrounded by excess HI absorption over the cosmic mean out to 4 cMpc/h. We also find that redshifts from the peak of the Ly$\alpha$ line are typically offset from the systemic redshift by +170 km/s. This work extends results from $z\approx 2 - 3$ to higher redshifts and demonstrates the power of deep integral field spectroscopy to simultaneously measure the ionization structure of the IGM and the large-scale density field in the early Universe.