Vacancy-ordered halide double perovskites (VODPs) have been widely explored throughout the past few years as promising lead-free alternatives for optoelectronic applications. Yet, the atomic-scale mechanisms that underlie their optical properties remain elusive. In this work, a thorough investigation of the excitonic properties of key members within the VODP family is presented. We employ state-of-the-art ab initio calculations and unveil critical details regarding the role of electron–hole interactions in the electronic and optical properties of VODPs. The materials family is sampled by picking prototypes based on the electronic configuration of the tetravalent metal at the center of the octahedron. Hence, groups with a valence comprised of s, p, and d closed shells are represented by the known materials Cs2SnX6, Cs2TeX6, and Cs2ZrX6 (with X = Br, I), respectively. The electronic structure is investigated within the G0W0 many-body Green’s function method, while the Bethe–Salpeter equation is solved to account for electron–hole interactions that play a crucial role in the optical properties of the family. A detailed symmetry analysis unravels the fine structure of excitons for all compounds. The exciton binding energy, excitonic wavefunctions, and the dark–bright splitting are also reported for each material. It is shown that these quantities can be tuned over a wide range, from Wannier- to Frenkel-type excitons, through for example substitutional engineering. In particular, Te-based materials, which share the electronic valency of corner-sharing Pb halide perovskites, are predicted to have exciton binding energies of above 1 eV and a dark–bright splitting of the excitons reaching over 100 meV. Our findings provide a fundamental understanding of the optical properties of the entire family of VODP materials and highlight how these are not, in fact, suitable Pb-free alternatives to traditional halide perovskites.