We consider the associated graded $\bigoplus_{k\geq 1} \Gamma_k \mathcal{I} / \Gamma_{k+1} \mathcal{I} $ of the lower central series $\mathcal{I} = \Gamma_1 \mathcal{I} \supset \Gamma_2 \mathcal{I} \supset \Gamma_3 \mathcal{I} \supset \cdots$ of the Torelli group $\mathcal{I}$ of a compact oriented surface. Its degree-one part is well-understood by D. Johnson's seminal works on the abelianization of the Torelli group. The knowledge of the degree-two part $(\Gamma_2 \mathcal{I} / \Gamma_3 \mathcal{I})\otimes \mathbb{Q}$ with rational coefficients arises from works of S. Morita on the Casson invariant and R. Hain on the Malcev completion of $\mathcal{I}$. Here, we prove that the abelian group $\Gamma_2 \mathcal{I} / \Gamma_3 \mathcal{I}$ is torsion-free, and we describe it as a lattice in a rational vector space. As an application, the group $\mathcal{I}/\Gamma_3 \mathcal{I}$ is computed, and it is shown to embed in the group of homology cylinders modulo the surgery relation of $Y_3$-equivalence.