The measurement of the $21$ cm signal from the Cosmic Dawn is a major goal for several existing and upcoming radio interferometers such as NenuFAR and the SKA. During this era before the beginning of the Epoch of Reionization, the signal is more difficult to observe due to brighter foregrounds but reveals additional information on the underlying astrophysical processes encoded in the spatial fluctuations of the spin temperature of hydrogen. To interpret future measurements, controlling the level of accuracy of the Lyman-$\alpha$ flux modelling is mandatory. In this work, we evaluate the impact of various approximations that exist in the main fast modelling approach compared to the results of a costly full radiative transfer simulation. The fast SPINTER code, presented in this work, computes the Lyman-$\alpha$ flux including the effect of wing scatterings for an inhomogeneous emissivity field, but assuming an otherwise homogeneous expanding universe. The LICORICE code computes the full radiative transfer in the Lyman-$\alpha$ line without any substantial approximation. We find that the difference between homogeneous and inhomogeneous gas density and temperature is very small for the computed flux. On the contrary, neglecting the effect of gas velocities produces a significant change in the computed flux. We identify the causes (mainly Doppler shifts due to velocity gradients) and quantify the magnitude of the effect in both an idealised setup and a realistic cosmological situation. We find that the amplitude of the effect, up to a factor of $\sim 2$ on the $21$ cm signal power spectrum on some scales (depending on both other model parameters and the redshift), can be easily discriminated with an SKA-like survey and already be approached, particularly for exotic signals, by the ongoing NenuFAR Cosmic Dawn Key Science Program.