This paper deals with PépiMep, a diagnosis system in school algebra. Our proposal to evaluate the students’ open-ended answers is based on a mixed theoretical and empirical approach. First, researchers in Math Education list different types of anticipated patterns of answers and the way to evaluate them. Then, this information is stored in an XML file used by the system to match a student’s input with an anticipated answer. Third, as it is impossible to anticipate every student’s answer, the system can improve: when an unknown form is detected, it is added to the XML file after expert inspection. Results from testing 360 students showed that, in comparison with human experts, PépiMep (1) was very effective in recognizing the different types of solutions when students’ input was an algebraic expression (2) but was less effective when students entered a reasoned response expressed by a mix of algebraic expressions and natural language utterances.