Obtaining an accurate theoretical description of the emergent phenomena in oxide heterostructures is a major challenge. Recently, intriguing paramagnetic spin and pinned orbital moments have been discovered by x-ray magnetic circular dichro\"ism measurements at the Cu $L_{2,3}$-edge of a tetragonal CuO/SrTiO$_3$ heterostructure. Using first principles calculations, we propose a scenario that explains both types of moments, based on the formation of oxygen vacancies in the TiO$_2$ interface layer. We show the emergence of a paramagnetic 2D electron gas hosted in the interface CuO layer. It is invisible at the Ti $L_{2,3}$-edge since the valence of the Ti atoms remains unchanged. Strong structural distortions breaking both the local and global fourfold rotation $C_4$ symmetries at the interface lead to the in-plane pinning of the Cu orbital moment close to the vacancy. Our results, and in particular the pinning of the orbital moment, may have implications for other systems, especially monoxide/dioxide interfaces with similar metal-oxygen bond length and weak spin-orbit coupling.