Dynamic macrostructural and microstructural changes take place from the mid-fetal stage to 2 years after birth. Delineating structural changes of the brain during early development provides new insights into the complicated processes of both typical development and the pathological mechanisms underlying various psychiatric and neurological disorders including autism, attention deficit hyperactivity disorder and schizophrenia. Decades of histological studies have identified strong spatial and functional maturation gradients in human gray and white matter. The recent improvements in magnetic resonance imaging (MRI) techniques, especially diffusion MRI (dMRI), relaxometry imaging, and magnetization transfer imaging (MTI) have provided unprecedented opportunities to non-invasively quantify and map the early developmental changes at whole brain and regional levels. Here, we review the recent advances in understanding early brain structural development during the second half of gestation and the first two postnatal years using modern MR techniques. Specifically, we review studies that delineate the emergence and microstructural maturation of white matter tracts, as well as dynamic mapping of inhomogeneous cortical microstructural organization unique to fetuses and infants. These imaging studies converge into maturational curves of MRI measurements that are distinctive across different white matter tracts and cortical regions. Furthermore, contemporary models offering biophysical interpretations of the dMRI-derived measurements are illustrated to infer the underlying microstructural changes. Collectively, this review summarizes findings that contribute to charting spatiotemporally heterogeneous gray and white matter structural development, offering MRI-based biomarkers of typical brain development and setting the stage for understanding aberrant brain development in neurodevelopmental disorders.