The temperature on Earth varied largely in the Pleistocene from cold glacials to interglacials of different warmths. To contribute to an understanding of the underlying causes of these changes we compile various environmental records (and model-based interpretations of some of them) in order to calculate the direct effect of various processes on Earth's radiative budget and, thus, on global annual mean surface temperature over the last 800,000 years. The importance of orbital variations, of the greenhouse gases CO2, CH4 and N2O, of the albedo of land ice sheets, annual mean snow cover, sea ice area and vegetation, and of the radiative perturbation of mineral dust in the atmosphere are investigated. Altogether we can explain with these processes a global cooling of 3.9 ± 0.8 K in the equilibrium temperature for the Last Glacial Maximum (LGM) directly from the radiative budget using only the Planck feedback that parameterises the direct effect on the radiative balance, but neglecting other feedbacks such as water vapour, cloud cover, and lapse rate. The unaccounted feedbacks and related uncertainties would, if taken at present day feedback strengths, decrease the global temperature at the LGM by −8.0 ± 1.6 K. Increased Antarctic temperatures during the Marine Isotope Stages 5.5, 7.5, 9.3 and 11.3 are in our conceptual approach difficult to explain. If compared with other studies, such as PMIP2, this gives supporting evidence that the feedbacks themselves are not constant, but depend in their strength on the mean climate state. The best estimate and uncertainty for our reconstructed radiative forcing and LGM cooling support a present day equilibrium climate sensitivity (excluding the ice sheet and vegetation components) between 1.4 and 5.2 K, with a most likely value near 2.4 K, somewhat smaller than other methods but consistent with the consensus range of 2–4.5 K derived from other lines of evidence. Climate sensitivities above 6 K are difficult to reconcile with Last Glacial Maximum reconstructions.