Research on the challenges of learning algebra is extensive. Various typical errors and misconcep-tions, like the ‘Letter as Object’ misconception, have been identified. Making use of such findings for formative assessment is the core idea of the SMART online tests (‘Specific Mathematics Assessments that Reveal Thinking’) that have been developed at the University of Melbourne. Adapting the test ‘Meaning of Letters’ for use in Germany, we are investigating if these multiple-choice items indeed reveal the misconceptions of German students adequately. Based on student interviews, we added two open-ended items to the test and analysed the written explanations of an initial sample of 600 German grade 7 and 8 students. The analysis shows a satisfactory concordance between the auto-matic SMART diagnosis and the student explanations, and that the ‘Letter as Object’ misconception was highly prevalent among those students.