We explore the statistical properties of the Casimir-Polder potential between a dielectric sphere and a three-dimensional heterogeneous medium, by means of extensive numerical simulations based on the scattering theory of Casimir forces. The simulations allow us to confirm recent predictions for the mean and standard deviation of the Casimir potential, and give us access to its full distribution function in the limit of a dilute distribution of heterogeneities. These predictions are compared with a simple statistical model based on a pairwise summation of the individual contributions of the constituting elements of the medium.