We aimed to examine the white matter changes associated with lexical production difficulties, beginning in midlife with increased naming latencies. To delay lexical production decline, middle-aged adults may rely on domain-general and language-specific compensatory mechanisms proposed by the LARA model (Lexical Access and Retrieval in Aging). However, the white matter changes supporting these mechanisms remains largely unknown. Using data from the CAMCAN cohort, we employed an unsupervised and data-driven methodology to examine the relationships between diffusion-weighted imaging and lexical production. Our findings indicate that midlife is marked by alterations in brain structure within distributed dorsal, ventral, and anterior cortico-subcortical networks, marking the onset of lexical production decline around ages 53–54. Middle-aged adults may initially adopt a “semantic strategy” to compensate for lexical production challenges, but this strategy seems compromised later (ages 55–60) as semantic control declines. These insights underscore the interplay between domain-general and language-specific processes in the trajectory of lexical production performance in healthy aging and hint at potential biomarkers for language-related neurodegenerative pathologies.