The Hartree-Fock ground state of the Homogeneous Electron Gas is never translation invariant, even at high densities. As proved by Overhauser, the (paramagnetic) free Fermi Gas is always unstable under the formation of spin or charge density waves. We give here the first explicit bound on the energy gain due to the breaking of translational symmetry. Our bound is exponentially small at high density, which justifies a posteriori the use of the non-interacting Fermi Gas as a reference state in the large-density expansion of the correlation energy of the Homogeneous Electron Gas. We are also able to discuss the positive temperature phase diagram and prove that the Overhauser instability only occurs at temperatures which are exponentially small at high density. Our work sheds a new light on the Hartree-Fock phase diagram of the Homogeneous Electron Gas.