We consider a second order elliptic PDE discretized by the Hybrid High Order (HHO) method, for which globally coupled unknowns are located at faces. To efficiently solve the resulting linear system, we propose a geometric multigrid algorithm that keeps the degrees of freedom on the faces at every level. The core of the algorithm resides in the design of the prolongation operator that passes information from coarse to fine faces through the reconstruction of an intermediary polynomial of higher degree on the cells. Higher orders are natively handled by the conservation of the same polynomial degree at every level. The proposed algorithm requires a hierarchy of nested meshes where the faces are also successively coarsened. Numerical tests on homogeneous and heterogeneous diffusion problems in square and cubic domains show fast convergence, scalability in the mesh size and polynomial order, and robustness with respect to heterogeneity of the diffusion coefficient.