Background: Brain white matter hyperintensities (WMH) are frequent in older adults, including cognitively unimpaired individuals and are associated to decreased cognitive performances. The prevalence of WMH and their links with cognitive performance and with cortical β-amyloid (Aβ) burden have been rarely assessed in younger adults. The aim of this study was to assess the impact of WMH in younger and older cognitively unimpaired adults on cognitive performances and the possible interaction with cortical β-amyloid (Aβ) burden. Method: Two hundred and seventy-two cognitively unimpaired adults from the community (from the IMAP study and the Age-Well trial) were enrolled and age-stratified in 86 younger adults (YA; ≤60 years old) and 187 older adults (OA; >60 years old). All participants underwent i) a comprehensive neuropsychological assessment allowing to obtain composite scores of episodic memory, processing speed, working memory, executive functions, and global cognition; ii) brain structural T1 and FLAIR MRI scans used for automatic segmentation of WMH volumes; and iii) a Florbetapir-PET scan to measure cortical Aβ. Within each group, the relationships of total and regional WMH to age and cognition were assessed, as well as the impact of cortical Aβ on these relationships. Result: WMH increased with age in both groups but more sharply in OA than YA, particularly in the parietal lobe. Higher WMH volume was associated to worse cognitive performances in both groups. Particularly, working memory, processing speed and global cognition were negatively associated with global and regional WMH in YA and executive functions were negatively associated to occipital WMH in OA. Cortical Aβ differentially impacts the links between WMH and cognition in YA versus OA. Conclusion: WMH volume increases with age and is negatively associated to cognitive performances including in YA. This study argues for the clinical relevance of WMH even in middle age.