With its context-independent rules valid in any setting, mathematics is considered to be the champion of abstraction, and for a long time human mathematical reasoning was thought to follow nothing but the laws of logic. However, the idea that mathematics is grounded in nature has gained traction over the past decades, and the context-independency of mathematical reasoning has come to be questioned. The thesis we defend concerns the role played by general, non-mathematical knowledge on individuals’ understanding of numerical situations. We propose that what we count has a crucial impact on how we count, in the sense that humans' representation of numerical information is dependent on the semantic context in which it is embedded. More specifically, we argue that daily-life knowledge about the entities described in an arithmetic word problem can shape its interpretation and influence the strategy used to solve it, even among experts. We investigate this thesis using behavioral and physiological data collected among diverse populations, ranging from 2nd graders to expert mathematicians and math teachers. We discuss the general educational implications of these effects, and we propose new directions for future research on this crucial issue.