The El Niño-Southern Oscillation (ENSO) represents the most consequential fluctuation of the global climate system, with dramatic societal and environmental impacts. Its general dynamics are reasonably well understood and they involve ocean-atmosphere interactions that modify the Walker circulation in the equatorial Pacific. However, some of its space-time features remain stubbornly elusive such as its event-to-event diversity and asymmetry. Here we show that the spatial shifting movements of the Walker circulation control the ENSO space-time complexity in a major way. We encapsulate these movements in a minimal model by simply reframing the general geometry of conventional recharge-discharge linear dynamics. With this modification alone, the model captures altogether the essential ingredients of ENSO diversity, asymmetry, nonlinearity, non-normality and leading principal components. It also manages to do so in substantially simpler fashion than previously considered, which improves our ability to address ENSO complexity. The present paradigm of spatial shifting participates towards a more unified understanding of the ENSO, with potentially far reaching implications down the lines of modelling and prediction.