In this paper, in the framework of the BMV (Biréfringence Magnétique du Vide) project, whose goal is the measurement of the vacuum linear magnetic birefringence, we study the optical noise affecting these kinds of experiments by detailing its sources, and we perform experimental measurements of all of them, clarifying that the limiting noise is a birefringence one.[graphic not available: see fulltext][graphic not available: see fulltext]