In the standard framework of self-consistent many-body perturbation theory, the skeleton series for the self-energy is truncated at a finite order $N$ and plugged into the Dyson equation, which is then solved for the propagator $G_N$. We consider two examples of fermionic models, the Hubbard atom at half filling and its zero space-time dimensional simplified version. First, we show that $G_N$ converges when $N\to\infty$ to a limit $G_\infty\,$, which coincides with the exact physical propagator $G_{\rm exact}$ at small enough coupling, while $G_\infty \neq G_{\rm exact}$ at strong coupling. This follows from the findings of [Phys. Rev. Lett. 114, 156402 (2015)] and an additional subtle mathematical mechanism elucidated here. Second, we demonstrate that it is possible to discriminate between the $G_\infty=G_{\rm exact}$ and $G_\infty\neq G_{\rm exact}$ regimes thanks to a criterion which does not require the knowledge of $G_{\rm exact}\,$, as proposed in [Phys. Rev. B 93, 161102 (2016)].