This is the third paper in a series of four in which we use space adiabatic methods in order to incorporate backreactions among the homogeneous and between the homogeneous and inhomogeneous degrees of freedom in quantum cosmological perturbation theory. In this paper we consider a particular kind of cosmological perturbation theory which starts from a gauge fixed version of General Relativity. The gauge fixing is performed using a material reference system called Gaussian dust. The resulting system has no constraints any more but possesses a physical Hamiltonian that drives the dynamics of both geometry and matter. As observable matter content we restrict to a scalar field (inflaton). We then explore the sector of that theory which is purely homogeneous and isotropic with respect to the geometry degrees of freedom but contains inhomogeneous perturbations up to second order of the scalar field. The purpose of this paper is to explore the quantum field theoretical challenges of the space adiabatic framework in a cosmological model of inflation which is technically still relatively simple. We compute the quantum backreaction effects from every energy band of the inhomogeneous matter modes on the evolution of the homogeneous geometry up to second order in the adiabatic parameter. These contributions turn out to be significant due to the infinite number of degrees of freedom and are very sensitive to the choice of Fock representation chosen for the inhomogeneous matter modes.