We propose a new variational framework to remove random-valued impulse noise from images. This framework combines, in the same energy, a non-local $L^p$ data term and a total variation regularization term. The non-local $L^p$ term is a weighted $L^p$ distance between pixels, where the weights depend on a robust distance between patches centered at the pixels. In a first part, we study the theoretical properties of the proposed energy, and we show how it is related to classical denoising models for extreme choices of the parameters. In a second part, after having explained how to numerically find a minimizer of the energy thanks to primal-dual approaches, we show extensive denoising experiments on various images and noise intensities. The denoising performances of the proposed methods are on par with state of the art approaches, and the remarkable fact is that, unlike other successful variational approaches for impulse noise removal, they do not rely on a noise detector.