The present study investigated the timecourse of the interactions between visual perceptual processes and conceptual knowledge while manipulating object-context semantic congruity. Participants categorized objects that were either congruent (e.g., a broom in a kitchen) or incongruent with their context (e.g., a tiger in a street) as natural or man-made, while we recorded event-related brain potentials (ERPs). Unlike previous work that found an effect of congruity starting at ~ 270 ms post-stimulus (Mudrik, Lamy, & Deouell, 2010), the earliest sign of reliable context congruity effect arose at ~ 170 ms in the present study. This finding suggests early interactions between perceptual and knowledge-based memory processes during a conceptual categorization task. The semantic congruity effect was also associated with N300 and N400 effects, reflecting object model selection and semantic processing stages, respectively. All together, these findings bear specific implications for the modeling of object recognition in scenes, and more broadly on the impact of memory on visual processing.