What information is used for sorting pictures of complex stimuli into categories? We applied a reverse correlation method to reveal the visual features mediating categorization in humans and baboons. Two baboons and 6 humans were trained to sort, by species, pictures of human and baboon faces on which random visual noise was superimposed. On ambiguous probe trials, a human-baboon morph was presented, eliciting ``human'' responses on some trials and ``baboon'' responses on others. The difference between the noise patterns that induced the two responses made explicit the information mediating the classification. Unlike the humans, the baboons based their categorization on information that closely matched that used by a theoretical observer responding solely on the basis of the pixel similarities between the probe and training images. We show that the classification-image technique and principal components analysis provide a method to make explicit the differences in the information mediating categorization in humans and animals.