Consider a deterministic self-adjoint matrix X_n with spectral measure converging to a compactly supported probability measure, the largest and smallest eigenvalues converging to the edges of the limiting measure. We perturb this matrix by adding a random finite rank matrix with delocalized eigenvectors and study the extreme eigenvalues of the deformed model. We give necessary conditions on the deterministic matrix X_n so that the eigenvalues converging out of the bulk exhibit Gaussian fluctuations, whereas the eigenvalues sticking to the edges are very close to the eigenvalues of the non-perturbed model and fluctuate in the same scale. We generalize these results to the case when X_n is random and get similar behavior when we deform some classical models such as Wigner or Wishart matrices with rather general entries or the so-called matrix models.