A growing literature has been showing a profound impact of alphabetization at several levels of the visual system, including the primary visual cortex (Szwed et al., 2014) and higher-order ventral and dorsal visual areas (Carreiras et al., 2009; Dehaene et al., 2010). Importantly, in typical alphabetization courses, learning to read is not isolated but instead combined with both learning to write and learning to segment the spoken language, relating all these different representations to each other. Indeed, learning to write and to pronounce the elementary sounds of language promotes additional mapping between the visual and motor systems by linking visual representations of letters and motor plans for handwriting and speech production. Thus, besides the already recognized influence of the phonological system, the potential influence from other neural systems in the functioning of the visual system seems to be relatively neglected. In this opinion paper we highlight the importance of multi-systems interplay during literacy acquisition, focusing on the question of how literacy breaks mirror invariance in the visual system. Specifically, we argue for a large contribution of top-down inputs from phonological, handwriting and articulatory representations toward the ventral visual cortex during the development of the visual word form system, which then plays a pivotal role in mirror discrimination of letters in literate individuals.