In speaker recognition, performance of a system is usually estimated globally on a large set of tests, even if it is well known that some subsets of tests could show a very different behavior from the complete set. In fact, a small subset of tests could represent the main part of the reported errors. In this work, we highlight a such subset of tests, for which impostors obtain very high recognition scores. We evaluate if the problem comes from the involved speakers, from the voice excerpts or from the client model estimation technique. We also propose a strategy in order to minimize the effects of the observed phenomena on the overall performance of the system.