Context. Rapidly rotating neutron stars are an ideal laboratory to test models of matter at high densities. In particular, the maximum rotation frequency of a neutron star depends on the equation of state and can be used to test models of the interior. However, observations of the spin distribution of rapidly rotating neutron stars show evidence for a lack of stars spinning at frequencies higher than f ≈ 700 Hz, well below the predictions of theoretical equations of state. This has generally been taken as evidence of an additional spin-down torque operating in these systems, and it has been suggested that gravitational wave torques may be operating and be linked to a potentially observable signal. Aims. We aim to determine whether additional spin-down torques (possibly due to gravitational wave emission) are necessary, or if the observed limit of f ≈ 700 Hz could correspond to the Keplerian (mass-shedding) break-up frequency for the observed systems, and is simply a consequence of the currently unknown state of matter at high densities.Methods. Given our ignorance with regard to the true equation of state of matter above nuclear saturation densities, we make a minimal physical assumption and only demand causality, that is, that the speed of sound in the interior of the neutron star should be lower than or equal to the speed of light c. We then connected our causally limited equation of state to a realistic microphysical crustal equation of state for densities below nuclear saturation density. This produced a limiting model that gave the lowest possible maximum frequency, which we compared to observational constraints on neutron star masses and frequencies. We also compared our findings with the constraints on the tidal deformability obtained in the observations of the GW170817 event.Results. We rule out centrifugal breakup as the mechanism preventing pulsars from spinning faster than f ≈ 700 Hz, as the lowest breakup frequency allowed by our causal equation of state is f ≈ 1200 Hz. A low-frequency cutoff, around f ≈ 800 Hz could only be possible when we assume that these systems do not contain neutron stars with masses above M ≈ 2 M⊙. This would have to be due either to selection effects, or possibly to a phase transition in the interior of the neutron star that leads to softening at high densities and a collapse to either a black hole or a hybrid star above M ≈ 2 M⊙. Such a scenario would, however, require a somewhat unrealistically stiff equation of state for hadronic matter, in tension with recent constraints obtained from gravitational wave observations of a neutron star merger.Key words: dense matter / stars: neutron / X-rays: binaries