We consider in this paper multiple 5G base stations (BSs) implementing Advanced Sleep Modes (ASM) wherein each base station is able to deactivate some of its components when it does not transport any traffic and save thus energy. Thanks to so-called lean carrier, ASM define four levels of sleep, the deeper the level the larger the energy gain but the more delay to wakeup and serve the incoming user. We specifically study this energy saving versus delay performance trade-off taking into account the effect of inter-cell interference and its impact on whether to wakeup and serve the transmission request immediately upon arrival or to continue to sleep; this latter decision is a main novelty of our work. We treat the case where arrivals of those requests are unknown and a reinforcement learning agent is implemented in each BS in order to (selfishly) derive the optimal sleep policy that achieves a target energy saving versus delay performance tradeoff. Our results show the optimal policies in terms of the value of the timer after which the BS goes into sleep, the time spent in each sleep level, and whether the BS should continue to sleep or wake up immediately upon request arrival. We eventually show the corresponding achieved power saving and delay performance. Index Terms—5G, multiple base stations, Advanced Sleep Modes, Multi-Agent Reinforcement Learning, energy saving versus delay performance trade-off.