Healthy aging is accompanied by heterogeneous decline of cognitive abilities among individuals, especially during senescence. The mechanisms of this variability are not understood, but have been associated with the reorganization of white matter fiber tracts and the functional co-activations of brain regions. Here, we built a causal inference framework to provide mechanistic insight into the link between structural connectivity and brain function, informed by brain imaging data and network modeling. By applying various degrees of interhemispheric degradation of structural connectivity, we were not only able to reproduce the age-related decline in interhemispheric functional communication and the associated dynamical flexibility, but we obtained an increase of global modulation of structural connectivity over the brain function during senescence. Notably, the increase in modulation between structural connectivity and brian function was higher in magnitude and steeper in its increase in older adults with poor cognitive performance. We independently validated the causal hypothesis of our framework via a Bayesian approach based on deep-learning. The current results might be the first mechanistic demonstration of dedifferentiation and scaffolding during aging leading to cognitive decline demonstrated in a large cohort.