Parallel transport is an important step in many discrete algorithms for statistical computing on manifolds. Numerical methods based on Jacobi fields or geodesics parallelograms are currently used in geometric data processing. In this last class, pole ladder is a simplification of Schild's ladder for the parallel transport along geodesics that was shown to be particularly simple and numerically stable in Lie groups. So far, these methods were shown to be first order approximations of the Riemannian parallel transport, but higher order error terms are difficult to establish. In this paper, we build on a BCH-type formula on affine connection spaces to establish the behavior of one pole ladder step up to order 5. It is remarkable that the scheme is of order three in general affine connection spaces with a symmetric connection, much higher than expected. Moreover, the fourth-order term involves the covariant derivative of the curvature only, which is vanishing in locally symmetric space. We show that pole ladder is actually locally exact in these spaces, and even almost surely globally exact in Riemannian symmetric manifolds. These properties make pole ladder a very attractive alternative to other methods in general affine manifolds with a symmetric connection.