Given an oriented surface bounding a handlebody, we study the subgroup of its mapping class group defined as the intersection of the handlebody group and the second term of the Johnson filtration: $\mathcal{A} \cap J_2$. We introduce two trace-like operators, inspired by Morita's trace, and show that their kernels coincide with the images by the second Johnson homomorphism $\tau_2$ of $J_2$ and $\mathcal{A} \cap J_2$, respectively. In particular, we answer by the negative to a question asked by Levine about an algebraic description of $\tau_2(\mathcal{A} \cap J_2)$. By the same techniques, and for a Heegaard surface in $S^3$, we also compute the image by $\tau_2$ of the intersection of the Goeritz group $\mathcal{G}$ with $J_2$.