We study by light microscopy a soft glass consisting of a compact arrangement of polydisperse elastic spheres. We show that its slow and non-stationary dynamics results from the unavoidable small fluctuations of temperature, which induce intermittent local mechanical shear in the sample, because of thermal expansion and contraction. Temperature-induced shear provokes both reversible and irreversible rearrangements whose amplitude decreases with time, leading to an exponential slowing down of the dynamics with sample age.