Patch models have proven successful to solve a variety of inverse problems in image restoration. Recent methods, combining Gaussian patch models with a Bayesian approach, achieve state-of-the-art results in several restoration problems. Different strategies are followed to define and estimate the patch models. In particular, a fixed model can be used for all image patches, or the model can be estimated locally for each patch. Per-patch model estimation has proven very powerful for image denoising, but it becomes seriously ill-posed for other inverse problems such as the interpolation of random missing pixels or zooming. In this work, we present a new framework for image restoration that makes it possible to use per-patch priors for these more general inverse problems. To this aim, we make use of a hyperprior on the model parameters which overcomes the ill-posedness of the per-patch estimation. We also make this framework general enough to include realistic additive noise models. This yields state-of-the-art results in problems such as interpolation, denoising and zooming. Moreover, taking advantage of the generality of the framework, we present an application to the generation of high dynamic range images from a single snapshot. Experiments conducted on synthetic and real data show the effectiveness of the proposed approach.