It was recently conjectured that in generic quantum many-body systems, the spectral density of local operators has the slowest high-frequency decay as permitted by locality. We show that the infinite-temperature version of this "universal operator growth hypothesis" holds for the quantum Ising spin model in $d \ge 2$ dimensions, and for the chaotic Ising chain (with longitudinal and transverse fields) in one dimension. Moreover, the disordered chaotic Ising chain that exhibits many-body localization can have the same high-frequency spectral density decay as thermalizing models. Our argument is statistical in nature, and is based on the observation that the moments of the spectral density can be written as a sign-problem-free sum over paths of Pauli string operators.