Let $U_h:\mathbb R^{d}\to \mathbb R^{d}$ be a smooth vector field and consider the associated overdamped Langevin equation $$dX_t=-U_h(X_t)\,dt+\sqrt{2h}\,dB_t$$ in the low temperature regime $h\rightarrow 0$. In this work, we study the spectrum of the associated diffusion $L=-h\Delta+U_h\cdot\nabla$ under the assumptions that $U_h=U_{0}+h\nu$, where the vector fields $U_{0}:\mathbb R^{d}\to \mathbb R^{d}$ and $\nu:\mathbb R^{d}\to \mathbb R^{d}$ are independent of $h\in(0,1]$, and that the dynamics admits $e^{-\frac Vh}$ as an invariant measure for some smooth function $V:\mathbb{R}^d\rightarrow\mathbb{R}$. Assuming additionally that $V$ is a Morse function admitting $n_0$ local minima, we prove that there exists $\epsilon>0$ such that in the limit $h\to 0$, $L$ admits exactly $n_0$ eigenvalues in the strip $\{\operatorname{Re}(z)< \epsilon\}$ which have moreover exponentially small moduli. Under a generic assumption on the potential barriers of the Morse function $V$, we also prove that the asymptotic behaviors of these small eigenvalues are given by Eyring-Kramers type formulas.