The Johnson-Morita theory is an algebraic approach to the mapping class group of a surface $\Sigma$, in which one considers its action on the successive nilpotent quotients of the fundamental group $\pi=\pi_1(\Sigma)$. In this paper, we develop an analogue of this theory for the handlebody group, i.e. the mapping class group of a handlebody $V$ bounded by a surface $\Sigma$, by considering its action on the pair $(\pi, \mathsf{A})$, where $\mathsf{A}$ denotes the kernel of the homomorphism induced by the inclusion of $\Sigma$ in $V$. We give a detailed study of the analogues of the Johnson filtration and the Johnson homomorphisms that arise in this new context. In particular, we obtain new representations of subgroups of the handlebody group into spaces of oriented trees with beads.