The Hofstadter problem is the lattice analog of the quantum Hall effect and is the paradigmatic example of topology induced by an applied magnetic field. Conventionally, the Hofstadter problem involves adding ∼104∼104\sim 10^4 T magnetic fields to a trivial band structure. In this work, we show that when a magnetic field is added to an initially topological band structure, a wealth of remarkable possible phases emerges. We prove that threading magnetic flux through a Hamiltonian with nonzero Chern number enforces a phase transition at fixed filling and that a 2D Hamiltonian with nontrivial Kane-Mele invariant produces a 3D TI or 3D weak TI phase in periodic flux. We then study fragile topology protected by the product of two-fold rotation and time-reversal and show that there exists a 3D higher order TI phase where corner modes are pumped by flux. We show that a model of twisted bilayer graphene realizes this phase. Our results rely primarily on the magnetic translation group which exists at rational values of the flux. The advent of Moir\'e lattices also renders our work relevant experimentally. In Moir\'e lattices, it is possible for fields of order 1−301−301-30 T to reach one flux per plaquette and allow access to our proposed Hofstadter topological phase.