This paper introduces semi-universal perturbations that bridge the gap between specific and universal adversarial perturbations. The original idea is to craft a specific perturbation by choosing it among a set of $L$ universal perturbations. We propose to jointly learn the perturbations of this set to maximize the chances to attack each example by allowing it to choose its own perturbation. To do so, we derive an algorithm, with convergence guarantees under Lipschitz continuity assumptions. Semi-universal perturbations offer a better flexibility, interpretability and diversity, confirmed by our experiments. Additionally, we provide a generalization bound on the abilities of the perturbations to attack new examples.