We investigate the origin of yield stress aging in semidense, saline, and turbid suspensions in which structural evolution is rapidly arrested by the formation of thermally irreversible roll-resisting interparticle contacts. By performing optical tweezer three-point bending tests on particle rods, we show that these contacts yield by overcoming a rolling threshold, the critical bending moment of which grows logarithmically with time. We demonstrate that this time-dependent contact-scale rolling threshold controls the suspension yield stress and its aging kinetics. We identify a simple constitutive relation between the contact-scale flexural rigidity and rolling threshold, which transfers to macroscopic scales. This leads us to establishing a constitutive relation between macroscopic shear modulus and yield stress that is generic for an array of colloidal systems.