Image restoration has come a long way since the early age of image processing. Deep learning methods nowadays give outstanding results, yet very few are actually used in digital illustration and photo retouching software due to large memory storage, massive computational requirements, but also the lack of user control and customization. This paper introduces a new lightweight framework for stylized and controlled image restoration using multi-scale networks built with independent parallel branches. The approach -based on two independent and complementary tasks- aims at: i. designing a lightweight network based on image processing techniques making it usable on light hardware architectures (low memory/computational costs); ii. providing a versatile, controllable and customizable network to stylize results in a plug-and-play manner. For various image restoration tasks (super-resolution, denoising, sharpening and inpainting), we demonstrate that the proposed method offers significant advantages over state-of-the-art reference-based approaches regarding these aspects.