Transmon qubits are ubiquitously used in quantum information processing architectures based on superconducting circuits. Because of their lifetime, strong drives are required to realize fast, high-fidelity, gates and measurements, including parametrically activated processes. Here, we show that even off-resonant drives, in regimes routinely used in experiments, can cause strong modifications to the structure of the transmon spectrum rendering a large part of it chaotic. Accounting for the full nonlinear dynamics in a Floquet-Markov formalism, we find that the coherence time of the computational states of transmons can be significantly altered in the presence of a chaotic layer. In particular, chaos-assisted quantum phase slips greatly enhance band dispersions. In the presence of a measurement resonator, we find that approaching chaotic behavior correlates with strong transmon-resonator hybridization, and an average resonator response centered on the bare resonator frequency. The phenomena described here should be present in all circuits based on low-impedance Josephson-junctions.