This paper presents a simple yet effective method to detect fake synthetic images generated by recent deep generative models, the so-called deepfakes. Unlike existing methods that require a relatively large number of real and fake training images, our method follows a novel idea of using only real images during the training phase. Our proposal is to construct proxy negative training samples, representing fake images, by applying an appropriate transformation on the real images in the training set. The training of our detector leverages the popular CLIP model as well as a center loss to encourage clustering of real images, with the aim of obtaining discriminative features for the classification of real and fake images. The proposed forensic detector is conceptually simple and data-efficient, i.e., it can be trained by using a small amount of only 4K real images. Experimental results and comparisons show the effectiveness of our method in terms of generalization capability to detect fake images generated by various deep generative models.