We develop a dynamic model of public debt under the assumption that it is problematic for governments to implement fast increases of tax revenues, as new taxes require costly infrastructure and expertise that can be built only over time. In this environment, the standard condition requiring economic growth greater than interest costs is not sufficient to guarantee financial stability. Debt might become unstable if the gap between these two indicators falls below a given threshold. Our empirical analysis based on historical public finance data for the US provides strong support for the model. This study conveys a cautionary warning, because the debt of relatively safe borrowers may suddenly become unstable for instance because of a substantial deceleration in the growth of nominal income. These issues can be particularly relevant for those countries that do not have a modern and efficient tax collection system.