Recent evidence suggests that a high glycemic load (GL) diet is a risk factor for dementia, especially among apolipoprotein E ε4 allele (APOE4) carriers, while its association with cognitive decline is poorly known. Here, we investigated the association of high-GL meals with cognitive decline in older adults during a 12-year follow-up, according to their APOE4 carrier status. We used random-effect models and data from 2539 elderly participants from the Three-City study who completed a food frequency questionnaire (FFQ) to longitudinally assess the association of GL with changes in different cognitive domains (verbal fluency, visual memory, attention, visual motor processing speed, episodic memory). In APOE4 carriers, afternoon snack with high GL was significantly associated with cognitive decline in visual memory, episodic memory, and global cognition compared with APOE4 non-carriers. This study suggests a detrimental association between a high-GL diet and cognitive decline. The promotion of a low GL diet as a target to prevent cognitive decline in high-risk populations deserves more research.