Understanding the algebraic structure underlying a manifold with a general affine connection is a natural problem. In this context, A. V. Gavrilov introduced the notion of framed Lie algebra, consisting of a Lie bracket (the usual Jacobi bracket of vector fields) and a magmatic product without any compatibility relations between them. In this work we will show that an affine connection with curvature and torsion always gives rise to a post-Lie algebra as well as a D-algebra. The notions of torsion and curvature together with Gavrilov's special polynomials and double exponential are revisited in this post-Lie algebraic framework. We unfold the relations between the post-Lie Magnus expansion, the Grossman-Larson product and the K-map, α-map and β-map, three particular functions introduced by Gavrilov with the aim of understanding the geometric and algebraic properties of the double-exponential, which can be understood as a geometric variant of the Baker-Campbell-Hausdorff formula. We propose a partial answer to a conjecture by Gavrilov, by showing that a particular class of geometrically special polynomials is generated by torsion and curvature. This approach unlocks many possibilities for further research such as numerical integrators and rough paths on Riemannian manifolds.