We easily recover the causal properties of visual events, enabling us to understand and predict changes in the physical world. We see a tennis racket hitting a ball and sense that it caused the ball to fly over the net; we may also have an eerie but equally compelling experience of causality if the streetlights turn on just as we slam our car's door. Both perceptual [1] and cognitive [2] processes have been proposed to explain these spontaneous inferences, but without decisive evidence one way or the other, the question remains wide open [3-8]. Here, we address this long-standing debate using visual adaptation-a powerful tool to uncover neural populations that specialize in the analysis of specific visual features [9-12]. After prolonged viewing of causal collision events called ``launches'' [1], subsequently viewed events were judged more often as noncausal. These negative aftereffects of exposure to collisions are spatially localized in retinotopic coordinates, the reference frame shared by the retina and visual cortex. They are not explained by adaptation to other stimulus features and reveal visual routines in retinotopic cortex that detect and adapt to cause and effect in simple collision stimuli.