This is the final paper in a series of four in which we incorporate backreaction among the homogeneous and between the homogeneous and inhomogeneous degrees of freedom in quantum cosmological perturbation theory using space adiabatic methods. Here, we consider the gauge-invariant scalar (Mukhanov-Sasaki) and tensor (primordial gravitational wave) perturbations of general relativity coupled to a scalar field which arise from a careful constraint analysis of this system up to second order in the perturbations. The simultaneous quantization of the homogeneous and inhomogeneous degrees of freedom requires the use of a suitable constructive perturbation scheme for which we employ space adiabatic perturbation theory, a rigorous extension of the standard Born-Oppenheimer theory. We are confronted with several challenges arising for theories with an infinite number of degrees of freedom. We are able to compute the backreaction effects up to second order in the adiabatic parameter and find modifications as compared to earlier derivations of the effective quantum dynamics of the homogeneous sector.