To survive in complex and seasonal environments, primates are thought to rely upon cognitive capacities such as decision-making and episodic memory, which enable them to plan their daily foraging path. According to the Ecological Brain hypothesis, feeding ecology has driven the expansion of the brain to support the corresponding development of cognitive skills. Recent works in cognitive neurosciences indicate that cognitive operations such as decision-making or subjective evaluation (which are contextual and dependent upon episodic memory), relied critically upon a small part of the frontal lobe, often referred to as the ventromedial prefrontal cortex (VMPFC). Several authors suggested that this area might be important for foraging, but this has never been tested. In the present study, we quantified the relation between the size of the VMPFC (along with other cerebral measures: the whole brain, the gyrus rectus and the somatosensory cortex) and key socio-ecological variables in five primate species (Macaca mulatta, Macaca fuscata, Gorilla gorilla, Pan troglodytes and Homo sapiens). We hypothesized that the size of the VMPFC would be greater in primates with a large dietary spectrum and complex foraging strategies. We also hypothesized that the impact of feeding ecology would be stronger on this specific region than on other regions (somatosensory cortex) or on more global cerebral measures (e.g., whole brain). In line with these hypotheses, we found that all cerebral measures were more strongly related to feeding ecology than group size, a proxy for social complexity. As expected, the VMPFC volume is more precisely related to feeding ecology than the whole brain, and appears to be critically related to dietary quality. Thus, combining a comparative approach with predictions coming both from behavioral ecology and cognitive neurosciences, our study provides evidence that feeding ecology played a key role in the development of specific cognitive skills, which rely upon the expansion of a specific cortical area.