We prove error estimates for the wave equation semi-discretized in space by the hybrid high-order (HHO) method. These estimates lead to optimal convergence rates for smooth solutions. We consider first the second-order formulation in time, for which we establish $H^1$ and $L^2$-error estimates, and the first-order formulation, for which we establish $H^1$-error estimates. For both formulations, the space semi-discrete HHO scheme has close links with hybridizable discontinuous Galerkin schemes from the literature. Numerical experiments using either the Newmark scheme or diagonally-implicit Runge-Kutta schemes for the time discretization illustrate the theoretical findings and show that the proposed numerical schemes can be used to simulate accurately the propagation of elastic waves in heterogeneous media.