Uganda universalized primary education in 1997. While this is indicative of government's will to democratize access to education, it was also characterized by an upsurge in primary school enrollments from slightly over 3 million in 1996 to close to 5.3 million in 1997. The primary education sector has equally been characterized by deteriorating indicators of internal efficiency, declining quality and dropouts. The 2014 population and housing census data reveals that about 6% of children aged 9-11 years had never been to school and that this proportion was disproportionately higher for some districts in north and north-eastern Uganda. We analysed the 10% sample of the 2014 census to try and shed light on the children aged 9-11 years that had never been to school. We used multilevel modelling with the district as contextual level of analysis to study the effect of individual, household and district level factors on the probability of never being enrolled. First and foremost, females were less likely to be out of school than their male counterparts, something novel given the existing literature on sex and access to education in Uganda. Controlling for individual level variations, we isolated district factors that are linked to non-enrollment and most importantly, found supply side factors like distance to school and pupil classroom ratio in a district to strongly influence school enrollment. Service provision is still very much at stake in many Ugandan districts.