What can false memories tell us about the structure of mental representations of arithmetic word problems? The semantic congruence model describes the central role of world semantics in the encoding, recoding, and solving of these problems. We propose to use memory tasks to evaluate key predictions of the semantic congruence model regarding the representations constructed when solving arithmetic word problems. We designed isomorphic word problems differing only by the world semantics imbued in their problem statement. Half the problems featured quantities (durations, heights, elevator floors) promoting an ordinal encoding, and the other half used quantities (weights, prices, collections) promoting a cardinal encoding. Across three experiments, in French and in English, we used surprise memory tasks to investigate adults’ mental representations when solving the problems. After the first solving task, the participants were given an unexpected task: either to recall the problems (Experiments 1 and 2) or to identify, from memory, the experimenter-induced changes in target problem sentences (Experiment 3). Crucially, all problems included a specific mathematical relationship that was not explicit in the problem statement and that could only be inferred from an ordinal encoding. We used the presence or absence of this relationship in the participants’ responses to infer the structure of their representations. Converging results from all three experiments bring new evidence of the role of semantic congruence in arithmetic reasoning, new insights into the relevance of the cardinal–ordinal distinction in numerical cognition, and a new perspective on the use of memory tasks to investigate variations in the representations of mathematical word problems.