Hybrid High-Order methods for elliptic diffusion problems have been originally formulated for loads in the Lebesgue space $L^2(\Omega)$. In this paper we devise and analyze a variant thereof, which is defined for any load in the dual Sobolev space $H^{-1}(\Omega)$. The main feature of the present variant is that its $H^1$-norm error can be bounded only in terms of the $H^1$-norm best error in a space of broken polynomials. We establish this estimate with the help of recent results on the quasi-optimality of nonconforming methods. We prove also an improved error bound in the $L^2$-norm by duality. Compared to previous works on quasi-optimal nonconforming methods, the main novelties are that Hybrid High-Order methods handle pairs of unknowns, and not a single function, and, more crucially, that these methods employ a reconstruction that is one polynomial degree higher than the discrete unknowns. The proposed modification affects only the formulation of the discrete right-hand side. This is obtained by properly mapping discrete test functions into $H^1_0(\Omega)$.