We consider a semiclassical linear Boltzmann model with a non local collision operator. We provide sharp spectral asymptotics for the small spectrum in the low temperature regime from which we deduce the rate of return to equilibrium as well as a metastability result. The main ingredients are resolvent estimates obtained via hypocoercive techniques and the construction of sharp Gaussian quasimodes through an adaptation of the WKB method.