We investigate the physics behind the complex thermo-mechanical behavior of clays. Depending on their loading history, clays exhibit thermal expansion or contraction, reversible or irreversible, and of much larger magnitude than for usual solids. This anomalous behavior is often attributed to water adsorption, but a proper link between adsorption and thermo-mechanics is still needed, which is the object of this paper. We propose a conceptual model starting from the scale of the adsorption up to the scale of the geomaterial, which successfully explains the thermo-mechanical behavior of clays. Adsorption takes place between clay layers at the nanometer scale. The mechanics of the clay layers is known to be strongly affected by adsorption, e.g., swelling with humidity increase. Here we investigate the effect of drained heating and show that an increase in temperature decreases the amplitude of the confining pressure oscillations with the basal spacing. More subtle is a shift of the oscillations to larger basal spacing. To relate the mechanics of a clay layer to that of the geomaterial, we propose an upscaling in two steps: the clay particle and the clay matrix with inclusions. We model the particle as a stack of layers in which different hydration states (number of water layers in a nanopore) can coexist. This description builds on the theory of shape memory alloys, the physics of which is quite analogous to the case of a clay particle. Upscaling to the scale of the clay matrix with inclusions is performed with conventional self-consistent homogenization. The conceptual model is confronted to three typical experiments of the thermo-mechanical behavior of clay. It captures all the anomalous behaviors of clays: expansion/contraction, reversibility/irreversibility, role of loading history, and impact on preconsolidation pressure. Moreover, it offers a possible nanoscale interpretation of each of these anomalous behaviors.