This paper is devoted to the linearized Vlasov-Poisson-Fokker-Planck system in presence of an external potential of confinement. We investigate the large time behaviour of the solutions using hypocoercivity methods and a notion of scalar product adapted to the presence of a Poisson coupling. Our framework provides estimates which are uniform in the diffusion limit. As an application in a simple case, we study the one-dimensional case and prove the exponential convergence of the nonlinear Vlasov-Poisson-Fokker-Planck system without any small mass assumption.