Cardiac electrophysiology simulations are numerically challenging due to the propagation of a steep electrochemical wave front and thus require discretizations with small mesh sizes to obtain accurate results. In this work, we present an approach based on the Hybridizable Discontinuous Galerkin method (HDG), which allows an efficient implementation of high-order discretizations into a computational framework. In particular using the advantage of the discontinuous function space, we present an efficient p-adaptive strategy for accurately tracking the wave front. HDG allows to reduce the overall degrees of freedom in the final linear system to those only on the element interfaces. Additionally, we propose a rule for a suitable integration accuracy for the ionic current term depending on the polynomial order and the cell model to handle high-order polynomials. Our results show that for the same number of degrees of freedom coarse high-order elements provide more accurate results than fine low-order elements. Introducing p-adaptivity further reduces computational costs while maintaining accuracy by restricting the use of high-order elements to resolve the wave front. For a patient-specific simulation of a cardiac cycle p-adaptivity reduces the average number of degrees of freedom by 95% compared to the non-adaptive model. In addition to reducing computational costs, using coarse meshes with our p-adaptive high-order HDG method also simplifies practical aspects of mesh generation and postprocessing.