We propose a general variance reduction strategy to compute averages with diffusion processes. Our approach does not require the knowledge of the measure which is sampled, which may indeed be unknown as for nonequilibrium dynamics in statistical physics. We show by a perturbative argument that a control variate computed for a simplified version of the model can provide an efficient control variate for the actual problem at hand. We illustrate our method with numerical experiments and show how the control variate is built in three practical cases: the computation of the mobility of a particle in a periodic potential; the thermal flux in atom chains, relying on a harmonic approximation; and the mean length of a dimer in a solvent under shear, using a non-solvated dimer as the approximation.