As introduced by [1], the privileged information is a complementary datum related to a training example that is unavailable for the test examples. In this paper, we consider the problem of recognizing low-resolution images (targeted task), while leveraging their high-resolution version as privileged information. In this context, we propose a novel framework for integrating privileged information in the learning phase of a deep neural network. We present a natural multi-class formulation of the addressed problem, while providing an end-to-end training framework of the internal deep representations. Based on a detailed analysis of the state-of-the-art approaches, we propose a novel loss function, combining two different ways of computing indicators of an example’s difficulty, based on its privileged information. We experimentally validate our approach in various contexts, proving the interest of our model for different tasks such as fine-grained image classification or image recognition from a dataset containing annotation noise.