We design and analyze a Hybrid High-Order (HHO) method on unfitted meshes to approximate elliptic interface problems by means of a consistent penalty methodà la Nitsche. The curved interface can cut through the mesh cells in a rather general fashion. Robustness with respect to the cuts is achieved by using a cell-agglomeration technique, and robustness with respect to the contrast in the diffusion coefficients is achieved by using a different gradient reconstruction on each side of the interface. A key novel feature of the gradient reconstruction is to incorporate a jump term across the interface, thereby releasing the Nitsche penalty parameter from the constraint of being large enough. Error estimates with optimal convergence rates are established. A robust cell-agglomeration procedure limiting the agglomerations to the nearest neighbors is devised. Numerical simulations for various interface shapes corroborate the theoretical results.