The diversity of cognitive skills across primates remains both a fascinating and a controversial issue. Recent comparative studies provided conflicting results regarding the contribution of social vs ecological constraints to the evolution of cognition. Here, we used an interdisciplinary approach combining comparative cognitive neurosciences and behavioral ecology. Using brain imaging data from 16 primate species, we measured the size of two prefrontal brain regions, the frontal pole (FP) and the dorso-lateral prefrontal cortex (DLPFC), respectively involved in metacognition and working memory, and examined their relation to a combination of socio-ecological variables. The size of these prefrontal regions, as well as the whole brain, was best explained by three variables: body mass, daily traveled distance (an index of ecological constraints) and population density (an index of social constraints). The strong influence of ecological constraints on FP and DLPFC volumes suggests that both metacognition and working memory are critical for foraging in primates. Interestingly, FP volume was much more sensitive to social constraints than DLPFC volume, in line with laboratory studies showing an implication of FP in complex social interactions. Thus, our data highlights the relative weight of social vs ecological constraints on the evolution of specific prefrontal brain regions and their associated cognitive operations in primates.