In this paper, we consider a ferromagnetic material of ellipsoidal shape. The associated magnetic moment then has two asymptotically stable opposite equilibria, of the form $\pm\overline{m}$. In order to use these materials for memory storage purposes, it is necessary to know how to control the magnetic moment. We use as a control variable a spatially uniform external magnetic field and consider the question of flipping the magnetic moment, i.e., changing it from the $+\overline{m}$ configuration to the $-\overline{m}$ one, in minimal time. Of course, it is necessary to impose restrictions on the external magnetic field used. We therefore include a constraint on the $L^\infty$ norm of the controls, assumed to be less than a threshold value $U$. We show that, generically with respect to the dimensions of the ellipsoid, there is a minimal value of $U$ for this problem to have a solution. We then characterize it precisely. Finally, we investigate some particular configurations associated to geometries enjoying symmetries properties and show that in this case the magnetic moment can be controlled in minimal time without imposing a threshold condition on $U$.