For any family of $N\times N$ random matrices $(\mathbf{A}_k)_{k\in K}$ which is invariant, in law, under unitary conjugation, we give general conditions for central limit theorems of random variables of the type $\operatorname{Tr}(\mathbf{A}_k \mathbf{M})$, where the Euclidean norm of $\mathbf{M}$ has order $\sqrt{N}$ (such random variables include for example the normalized matrix entries $\sqrt{N} \mathbf{A}_k(i,j)$). A consequence is the asymptotic independence of the projection of the matrices $\mathbf{A}_k$ onto the subspace of null trace matrices from their projections onto the orthogonal of this subspace. This result is used to study the asymptotic behaviour of the outliers of a spiked elliptic random matrix. More precisely, we show that their fluctuations around their limits can have various rates of convergence, depending on the Jordan Canonical Form of the additive perturbation. Also, some correlations can arise between outliers at a macroscopic distance from each other. These phenomena have already been observed by Benaych-Georges and Rochet with random matrices from the Single Ring Theorem.