Abstract Introduction White matter (WM) analysis in neonatal brain magnetic resonance imaging (MRI) is challenging, as demonstrated by the issue of diffuse excessive high signal intensity (DEHSI). We evaluated the reliability of the radiologist'eye in this context. Methods Three experienced observers graded the WM signal intensity on axial T2-weighted 1.5T images from 60 different premature newborns on 2 occasions 4 weeks apart with a semi-quantitative classification under identical viewing conditions. Results The intra- and inter-observer correlation coefficients were fair to moderate (Fleiss’kappa between 0.21 and 0.60). Conclusion This is a serious limitation of which we need to be aware, as it can lead to contradictory conclusions in the challenging context of term-equivalent age brain MRI in premature infants. These results highlight the need for a semiautomatic tool to help in objectively analyzing MRI signal intensity in the neonatal brain.