The left ventral occipito-temporal cortex (vOT) is the key area of the visuo-orthographic system. However, some studies also reported its activation during speech processing tasks, especially those that rely on metaphonological or orthographic knowledge. These findings suggest such cross-modal activation is supported by top-down mechanisms. Yet, little is known about the involvement of the vOT during natural speech processing situations. Based on the assumption that top-down activation of the vOT is more likely to occur in demanding speech processing situations, we conducted an fMRI experiment manipulating two factors: (1) task demands (semantic vs. low-level perceptual task), and (2) the quality of the spoken input (sentences were clearly presented vs. degraded by conversation noise background). Results revealed vOT activation in all experimental conditions, at the same location as the one observed during visual word processing. Interestingly, the level of activation significantly increased in the semantic compared to perceptual task. However, in contrast to our expectation, while the degradation of the spoken input by conversation noise tended to increase activation in the superior temporal or inferior frontal cortex, reflecting a greater difficulty to process a noisy signal, it significantly reduced the activation of the vOT observed in the semantic task. We concluded that the involvement of the visuo-orthographic system during speech processing is supported by two complementary mechanisms. First, a task-dependent mechanism that recruits all available linguistic information during high-level language processing. Second, spontaneous, stimulus-driven activation of orthographic representations by spoken input, which is diminished in degraded hearing conditions.