In this paper, we introduce a new definition of the Koopman operator which faithfully encodes the dynamics of controlled systems, by leveraging the grammar of set-valued analysis. We likewise propose meaningful generalisations of the Liouville and Perron-Frobenius operators, and show that they respectively coincide with proper set-valued analogues of the infinitesimal generator and dual operator of the Koopman semigroups. We also give meaning to the spectra of these set-valued operators and prove an adapted version of the spectral mapping theorem. In essence, these results provide theoretical justifications for many existing approaches that consist in bundling together the Liouville operators associated with different control parameters to produce Koopman eigenvalues and eigenfunctions for control systems.