Shortly after reading instruction, a region in the ventral occipital temporal cortex (vOTC) of the left hemisphere, the Visual Word Form Area (VWFA), becomes specialized for written words. Its reproducible location across scripts suggests important anatomical constraints, such as specific patterns of connectivity, notably to spoken language areas. Here, we explored the structural connectivity of the emerging VWFA in terms of its specificity relative to other ventral visual regions and its stability throughout the process of reading instruction in 10 children studied longitudinally over 2 years. Category-specific regions for words, houses, faces, and tools were identified in the left vOTC of each subject with functional MRI. With diffusion MRI and tractography, we reconstructed the connections of these regions at two time points (mean age ± standard deviation: 6.2±0.3, 7.2±0.4 years). We first showed that the regions for each visual category harbor their own specific connectivity, all of which precede reading instruction and remain stable throughout development. The most specific connections of the VWFA were to the dorsal posterior parietal cortex. We then showed that microstructural changes in these connections correlated with improvements in reading scores over the first year of instruction but not one year later in a subsample of 8 children (age: 8.4±0.3 years). These results suggest that the VWFA location depends on its connectivity to distant regions, in particular the left inferior parietal region which may play a crucial role in visual field maps and eye movement dynamics in addition to attentional control in letter-by-letter reading and disambiguation of mirror letters during the first stages of learning to read.