We investigate the standard deviation δv(∆t) of the variance v[x] of time series x measured over a finite sampling time ∆t focusing on non-ergodic systems where independent "configurations" c get trapped in meta-basins of a generalized phase space. It is thus relevant in which order averages over the configurations c and over time series k of a configuration c are performed. Three variances of v[x ck ] must be distinguished: the total variance δv 2 tot = δv 2 int +δv 2 ext and its contributions δv 2 int , the typical internal variance within the meta-basins, and δv 2 ext , characterizing the dispersion between the different basins. We discuss simplifications for physical systems where the stochastic variable x(t) is due to a density field averaged over a large system volume V. The relations are illustrated for the shear-stress fluctuations in quenched elastic networks and low-temperature glasses formed by polydisperse particles and free-standing polymer films. The different statistics of δvint and δvext are manifested by their different system-size dependences.