Medieval historical monuments of the Provence region (South East of France) were erected with bioclastic limestones and display different sensitivities to spalling decay. The present study aims at understanding the physical processes at play as well as the internal properties governing its intensity. Limestones of contrasting sensitivity to spalling were compared to a reference type, unaffected by this decay, by means of petrography, petrophysic, mineralogy, and hydromechanics. The obtained results highlighted that the various sensitivities can be explained by the deformation recorded during water content variation (hydric dilation). A clay fraction was systematically detected within the mineralogical composition except for the reference material, and some swelling layers were identified in ontmorillonite/glauconite mixed layer minerals. A specific quantification procedure based on the combination of transmission electron microscopy coupled to an energy-dispersive X-ray spectrometer (TEM–EDX) and profile modeling of X-ray diffraction patterns was applied. A strong relationship between swelling layer content and hydric dilation of limestones was evidenced and corroborated the spalling sensitivity. Further interpretation of results showed that swelling layers localization within the texture significantly influence hydric dilation kinetics. Eventually, a mechanical softening was measured after water saturation. This behavior seems unrelated to the clay mineral content and its relative influence on spalling should be examined.