We use H.E.S.S. $\gamma$-ray observations of Sgr A* to derive novel limits on the Dark Matter (DM) annihilation cross-section. We quantify their dependence on uncertainties i) in the DM halo profile, which we vary from peaked to cored, and ii) in the shape of the DM spike around Sgr A*, dynamically heated by the nuclear star cluster. For peaked halo profiles and depending on the heating of the spike, our limits are the strongest existing ones for DM masses above a few TeV. Our study contributes to assessing the influence of the advancements in our knowledge of the Milky Way on determining the properties of DM particles.