The change of physical conditions across the turbulent and magnetized interstellar medium (ISM) induces a 3D spatial variation of the properties of Galactic polarized emission. The observed signal results from the averaging of different spectral energy distributions (SED) and polarization angles, along and between lines of sight. As a consequence, the total Stokes parameters $Q$ and $U$ will have different distorted SEDs, so that the polarization angle becomes frequency dependent. In the present work, we show how this phenomenon similarly induces a different distorted SED for the three polarized angular power spectra $EE$, $BB$ and $EB$, implying a variation of the $EE/BB$ ratio with frequency. We demonstrate how the previously introduced spin-moment formalism provides a natural framework to grasp these effects, allowing us to derive analytical predictions for the spectral behaviors of the polarized spectra, focusing here on the example of thermal dust polarized emission. After a quantitative discussion based on a model combining emission from a filament with its background, we further reveal that the spectral complexity implemented in the dust models commonly used by the cosmic microwave background (CMB) community produce such effects. This new understanding is crucial for CMB component separation, in which an extreme accuracy is required in the modeling of the dust signal to allow for the search of the primordial imprints of inflation or cosmic birefringence. For the latter, as long as the dust $EB$ signal is not measured accurately, great caution is required about the assumptions made to model its spectral behavior, as it may not simply follow from the other dust angular power spectra.