Reading relies on the ability to map written symbols with speech sounds. The left ventral occipitotemporal cortex (left-vOT) plays a crucial role in this process. Through the automatization of the mapping ability, this specific part of the ventral visual pathway (a.k.a., the Visual Word Form Area) progressively becomes specialized in written word recognition. Yet, despite its key role in reading, the area also responds to speech. This observation raises questions about the actual nature of neural representations encoded in the left-vOT and, therefore, the underlying mechanism of the cross-modal responses. Here, we addressed this issue by applying fine-grained analyses of within-and cross-modal repetition suppression effects (RSEs) and Multi-Voxel Pattern Analyses in fMRI and sEEG experiments. Convergent evidence across analysis methods and protocols showed significant RSEs and successful decoding in both within-modal visual and auditory conditions suggesting that subpopulations of neurons within the left-vOT distinctly encode written and spoken language inputs. This functional organization of neural populations enables the area to respond to speech input directly and indirectly, i.e., after speech sounds are converted to orthographic representations. The finding opens further discussions on how the human brain may be prepared and adapted for an acquisition of a complex ability such as reading.