We study in three dimensions the problem of a spatially homogeneous zero-temperature ideal Fermi gas of spin-polarized particles of mass $m$ perturbed by the presence of a single distinguishable impurity of mass $M$. The interaction between the impurity and the fermions involves only the partial $s$-wave through the scattering length $a$, and has negligible range $b$ compared to the inverse Fermi wave number $1/\kf$ of the gas. Through the interactions with the Fermi gas the impurity gives birth to a quasi-particle, which will be here a Fermi polaron (or more precisely a {\sl monomeron}). We consider the general case of an impurity moving with wave vector $\KK\neq\OO$: Then the quasi-particle acquires a finite lifetime in its initial momentum channel because it can radiate particle-hole pairs in the Fermi sea. A description of the system using a variational approach, based on a finite number of particle-hole excitations of the Fermi sea, then becomes inappropriate around $\KK=\mathbf{0}$. We rely thus upon perturbation theory, where the small and negative parameter $\kf a\to0^-$ excludes any branches other than the monomeronic one in the ground state (as e.g.\ the dimeronic one), and allows us a systematic study of the system. We calculate the impurity self-energy $\Sigma^{(2)}(\KK,\omega)$ up to second order included in $a$. Remarkably, we obtain an analytical explicit expression for $\Sigma^{(2)}(\KK,\omega)$ allowing us to study its derivatives in the plane $(K,\omega)$. These present interesting singularities, which in general appear in the third order derivatives $\partial^3 \Sigma^{(2)}(\KK,\omega)$. In the special case of equal masses, $M=m$, singularities appear already in the physically more accessible second order derivatives $\partial^2 \Sigma^{(2)}(\KK,\omega)$; using a self-consistent heuristic approach based on $\Sigma^{(2)}$ we then regularise the divergence of the second order derivative $\partial_K^2 \Delta E(\KK)$ of the complex energy of the quasi-particle found in reference [C. Trefzger, Y. Castin, Europhys. Lett. {\bf 104}, 50005 (2013)] at $K=\kf$, and we predict an interesting scaling law in the neighborhood of $K=\kf$. As a by product of our theory we have access to all moments of the momentum of the particle-hole pair emitted by the impurity while damping its motion in the Fermi sea, at the level of Fermi's golden rule.