In this work, we present a fast direct solver for integral equations-based electromagnetic scattering that is compatible with spatial discretization of arbitrary order. The proposed approach leverages a suitably preconditioned combined field formulation and involves the regularization of the elliptic spectrum through the extraction of a specifically chosen equivalent circulant problem. The procedure yields a compressed representation of the inverse system matrix in the form of a matrix amenable to fast inversion and a single-skeleton low-rank perturbation. In addition to proposing the fast direct solution framework, this contribution presents the algorithms and explores the intricacies that enable the high-order discretization of the underlying integral equations and is supported by numerical results illustrating its effectiveness.