The crystal structure of U4O9 remains an enigma because of its differences with U4+ and U5+ coordination polyhedral mixtures, as shown in the XANES experimental results. To better understand this crystal structure, its diffraction pattern was measured at seven different temperatures using neutron diffraction before being independently refined by Rietveld’s method and pair distribution function analysis. The O cuboctahedron—a structural element consisting of 13 oxygen atoms—is a specific feature of the U4O9 crystal structure. The volume of the cuboctahedron decreases when the temperature increases, whereas the overall volume of the crystal cell increases. This feature can be correlated with the two U4O9 phase transitions that induce sharp changes in the cuboctahedron geometry, suggesting that this structural element has internal dynamics. In particular, these structural modifications in the γ phase suggest that the high-temperature phase can be described as a mixture of U4+ and U5+ coordination polyhedra, the latter having U–O distances shorter than 2.2 Å, that are absent in the former. These changes in uranium polyhedra as a function of temperature are tentatively interpreted using steric arguments. They also raise the question of charge localization on the different U ion sites in the low-temperature phases of U4O9.