We introduce a notion of geodesic curvature $k_{\zeta}$ for a smooth horizontal curve $\zeta$ in a three-dimensional contact sub-Riemannian manifold, measuring how much a horizontal curve is far from being a geodesic. We show that the geodesic curvature appears as the first corrective term in the Taylor expansion of the sub-Riemannian distance between two points on a unit speed horizontal curve $$ d_{SR}^2( \zeta(t),\zeta(t+\epsilon))=\epsilon^2-\frac{k_{\zeta}^2(t)}{720} \epsilon^6 +o(\epsilon^{6}). $$ The sub-Riemannian distance is not smooth on the diagonal, hence the result contains the existence of such an asymptotics. This can be seen as a higher-order differentiability property of the sub-Riemannian distance along smooth horizontal curves. It generalizes the previously known results on the Heisenberg group.