The relation between inflammation and brain MRI findings in the elderly remains poorly known. We investigated the association of circulating interleukin-6 (IL-6) and C-reactive protein (CRP) levels with baseline and longitudinal white matter hyperintensities (WMH), silent brain infarction, and brain volumes in community-dwelling elderly free of dementia. We included 1,841 participants aged 65 to 80 years from the Three City-Dijon cohort. Participants followed an MRI examination at baseline and after a 4-year follow-up (n = 1,316). IL-6 and CRP concentrations were measured at baseline from fasting blood samples. WMH were detected with an automatic imaging processing method and gray matter, hippocampal, white matter, and CSF volumes were estimated with voxel-based morphometry. Silent brain infarctions were assessed visually and defined as focal lesions of ≥3 mm in the absence of stroke. We used analysis of covariance and logistic regression to model the associations between inflammatory biomarkers and brain MRI findings adjusting for potential confounders. In cross-sectional analyses, higher IL-6 levels were associated with higher WMH volumes (p < 0.01), lower gray matter (p = 0.001) and hippocampal (p = 0.01) volumes, and increasing CSF volumes (p = 0.002) in a dose-relationship pattern. Similar but weaker relations were observed for CRP. We observed no associations between baseline inflammatory biomarker levels and the evolution of MRI findings over 4 years. IL-6, and, to a lesser degree, CRP levels were associated with WMH severity as well as global markers of brain atrophy. These results suggest that an inflammatory process may be involved in both age-associated brain alterations.