Cultural and geographical properties of the environment have been shown to deeply influence cognition and mental health. However, how the environment experienced during early life impacts later cognitive abilities remains poorly understood. Here, we used a cognitive task embedded in a video game to measure non-verbal spatial navigation ability in 442,195 people from 38 countries across the world. We found that on average, people who reported having grown up in cities have worse navigation skills than those who grew-up outside cities, even when controlling for age, gender, and level of education. The negative impact of cities was stronger in countries with low average Street Network Entropy, i.e. whose cities have a griddy layout. The effect was smaller in countries with more complex, organic cities. This evidences the impact of the environment on human cognition on a global scale, and highlights the importance of urban design on human cognition and brain function.