We simulate the propagation of cosmic rays at ultra-high energies, $\gtrsim 10^{18}$ eV, in models of extragalactic magnetic fields in constrained simulations of the local Universe. We investigate the impact of different magneto-genesis scenarios, both, primordial and astrophysical, on the propagation of cosmic rays. Our study shows that different scenarios of magneto-genesis do not have a large impact on the anisotropy measurements. The distribution of nearby sources causes anisotropy at very high energies, independent of the magnetic field model. We compare our results to the dipole signal measured by the Pierre Auger Observatory. All our models could reproduce the observed dipole amplitude with a pure iron injection composition. This is due to clustering of secondary nuclei in direction of nearby sources of heavy nuclei. A light injection composition is disfavoured by the non-observation of anisotropy at energies of 4 - 8 EeV.