We introduce a simple determinant diagrammatic Monte Carlo algorithm to compute the ground-stateproperties of a particle interacting with a Fermi sea through a zero-range interaction. The fermionic sign doesnot cause any fundamental problem when going to high diagram orders, and we reach order N = 30. The datareveal that the diagrammatic series diverges exponentially as (−1/R) N with a radius of convergence R < 1.Furthermore, on the polaron side of the polaron-dimeron transition, the value of R is determined by a specialclass of three-body diagrams, corresponding to repeated scattering of the impurity between two particles ofthe Fermi sea. A power-counting argument explains why finite R is possible for zero-range interactions in threedimensions. Resumming the divergent series through a conformal mapping yields the polaron energy with recordaccuracy.