Abstract The quiet Sun observed in polarized light exhibits a rich and complex magnetic structuring which is still not fully resolved nor understood. The present work is intended to contribute to the debate about the origin of the quiet sun magnetic fields, in relation or not to the global solar dynamo. We present analysis of center-to-limb polarization measurements obtained with the SOT/SP spectropolarimeter onboard the Hinode satellite outside active regions, in 2007 and 2013, i.e. at a minimum and a maximum of the solar cycle, respectively. We compare the spatial fluctuation Fourier spectra of unsigned circular and linear polarization images after corrections for polarization bias and focus variations between the two data sets. The decay of active regions is clearly a source of magnetic fields in the quiet Sun. It leads to a global increase of the polarization fluctuation power spectrum in 2013 in the network. In the internetwork, we observe no variation of the polarization fluctuation power at mesogranular and granular scales, whereas it increases at sub-granular scales. We interpret these results in the following way. At the mesogranular and granular scales very efficient mechanisms of magnetic field removal are operating in the internetwork, that leads to a dissipation or a concentration of magnetic fields on smaller scales. So the cycle-invariant magnetic signal that we detect at mesogranular and granular scales must be continuously created by a dynamo mechanism which is independent of the solar cycle.