Consider a deterministic self-adjoint matrix $X_n$ with spectral measure converging to a compactly supported probability measure, the largest and smallest eigenvalues converging to the edges of the limiting measure. We perturb this matrix by adding a random finite rank matrix with delocalized eigenvectors and study the extreme eigenvalues of the deformed model. We show that the eigenvalues converging out of the bulk exhibit Gaussian fluctuations, whereas under additional hypotheses, the eigenvalues sticking to the edges are very close to the eigenvalues of the non-perturbed model and fluctuate in the same scale. We can also generalize those results to the case when $X_n$ is random and get similar behavior when we deform some classical models such as Wigner or Wishart matrices with rather general entries or the so-called matrix models.