We consider the diffractive photo-production of vector mesons on a proton, in the dipole model. We take into account the effect of the fluctuations of the the dipole size, whose magnitude is controlled by the overlap between the photon and the vector meson wave functions. Our predictions for the incoherent diffractive cross section, obtained within the Impact Parameter Saturation Model (IPSat), are shown to be in excellent agreement with the HERA data on $J/\Psi$ photo-production, down to very low momentum transfer, where the dipole size fluctuations are shown to play an essential role. This study complements, without introducing any additional parameter, previous treatments of incoherent diffractive processes in terms of fluctuations of the proton shape, by adding another source of ``geometrical'' fluctuations, namely those coming from the splitting of the photon into color dipoles of various sizes.