We study the statistical fluctuations of the Casimir potential felt by an atom approaching a dielectric disordered medium. Starting from a microscopic model for the disorder, we calculate the variance of potential fluctuations in the limit of a weak density of heterogeneities. We show that fluctuations are essentially governed by scattering of the radiation on a single heterogeneity, and discuss in which limits they become larger than the average value predicted by effective medium theory. Finally, for denser disorder we show that multiple scattering processes become relevant.