In this paper, we build on optimal transport (OT) theory to present a novel asymmetrically unbalanced variant, the semi-unbalanced optimal transport (SUOT), specifically designed for imaging applications with the presence of a reference. SUOT addresses the lack of robustness of OT and the rigidity inherited from its formulation by taking inspiration from the unbalanced OT formulation. Rather than relaxing the constraints on both the source and the target measures, we relax only the marginal related to the reference. We consider both the unregularized and entropy-regularized versions, deriving dual formulations, corresponding minimization algorithms and formulas for the gradient. These derivations enable us to employ SUOT in variational inverse imaging and synthesis problems, as well as a loss for training a neural network. We evaluate the use of SUOT in a reference-driven super-resolution problem and show its benefits. We also incorporate it into a state-of-the-art single-image generation algorithm and show that it leads to increased diversity. Our results advocate for the adoption of SUOT as a general tool for variational and learning-based inverse imaging and synthesis problems with the presence of a reference.