We outline the key-steps towards the construction of a physical, fully relativistic cosmology, in which we aim to trace Dark Energy and Dark Matter back to physical properties of space. The influence of inhomogeneities on the effective evolution history of the Universe is encoded in backreaction terms and expressed through spatially averaged geometrical invariants. These are absent and interpreted as missing dark fundamental sources in the standard model. In the inhomogeneous case they can be interpreted as energies of an emerging scalar field (the morphon). These averaged invariants vanish for a homogeneous geometry, where the morphon is in an unstable equilibrium state. If this state is perturbed, the morphon can act as a classical inflaton in the Early Universe and its de-balanced energies can mimic the dark sources in the Late Universe, depending on spatial scale as Dark Energy or as Dark Matter, respectively. We lay down a line of arguments that is qualitatively conclusive, and we outline open problems of quantitative nature, related to the interpretation of observations.