Larger comparative studies of mathematics classrooms became most prominent as an appendix of international achievement studies. With the advance of video technology and the potentials it offers for researchers, however, comparative classroom observation studies became attractive and feasible also for smaller scale and low budget projects. This paper intends to provide a basis for discussing rationales for comparative studies of mathematics classrooms. It is suggested that the affiliation of classroom observation with school inspection on the one hand, and ethnographic research on the other hand, lingers on in comparative classroom research. The paper provides a narrative, illustrated by examples, that exposes this tension between evaluation and documentation inherent in the field.