The El Niño-Southern Oscillation (ENSO) phenomenon is the strongest mode of interannual climate variability, causing major disruptions of the tropical ocean-atmosphere system that affect the entire planet. Predicting El Niño occurrence and amplitude, both on the seasonal time scale and for the next century, is a key societal need and remains a major research challenge. Coupled ocean-atmosphere GCMs still exhibit errors in reproducing the observed amplitude, structure and frequency of El Niño events, thus limiting our ability to describe future changes in ENSO properties. Several studies using these models suggest that the atmospheric component plays a dominant role in setting the modeled ENSO characteristics, which today exhibit an inter-model diversity well beyond that observed. We here focus on the atmosphere's role in that diversity by analyzing the tropical Pacific ocean-atmosphere feedbacks in the IPCC AR4 multi-model database. Two main feedbacks are studied: the dynamical SST-wind stress (Bjerknes) positive feedback and the thermodynamical SST-heat flux negative feedback. It is shown that the models generally underestimate both of these, leading to a compensation of errors. It is further shown that most of the diversity among models comes from the shortwave radiation feedback. This feedback is tightly coupled to the ability of models to simulate the deep convection and subsidence regimes, two large-scale regimes for which current models still exhibit significant errors, especially in the equatorial eastern Pacific where ENSO is amplified.