Correlation functions of components of second-order tensor fields in isotropic systems can be reduced to an isotropic forth-order tensor field characterized by a few invariant correlation functions (ICFs). It is emphasized that components of this field depend in general on the coordinates of the field vector variable and thus on the orientation of the coordinate system. These angular dependencies are distinct from those of ordinary anisotropic systems. As a simple example of the procedure to obtain the ICFs we discuss correlations of time-averaged stresses in isotropic glasses where only one ICF in reciprocal space becomes a finite constant e for large sampling times and small wavevectors. It is shown that e is set by the typical size of the frozen-in stress components normal to the wavevectors, i.e. it is caused by the symmetry breaking of the stress for each independent configuration. Using the presented general mathematical formalism for isotropic tensor fields this finding explains in turn the observed long-range stress correlations in real space. Under additional but rather general assumptions e is shown to be given by a thermodynamic quantity, the equilibrium Young modulus E. We thus relate for certain isotropic amorphous bodies the existence of finite Young or shear moduli to the symmetry breaking of a stress component in reciprocal space.