We consider a second order elliptic PDE discretized by the Hybrid High-Order method, for which globally coupled unknowns are located at faces. To efficiently solve the resulting linear system, we propose a geometric multigrid algorithm that keeps the degrees of freedom on the faces at every grid level. The core of the algorithm lies in the design of the prolongation operator that passes information from coarse to fine faces through the reconstruction of an intermediary polynomial of higher degree on the cells. High orders are natively handled by the use of the same polynomial degree at every grid level. The proposed algorithm requires a hierarchy of meshes, non necessarily nested, such that the faces (and not only the elements) are successively coarsened. Numerical tests on homogeneous and heterogeneous diffusion problems show fast convergence, scalability in the mesh size and polynomial order, and robustness with respect to heterogeneity of the diffusion coefficient.