Advanced Sleep Modes (ASMs) are defined as a progressive shutdown of the Base Station (BS) depending on the activation and the deactivation times of the different components. This transition duration defines different levels of sleep modes that can be implemented in future 5G networks. We propose in this paper a management strategy based on Q-learning approach which will enable to find the best combination and durations of ASM levels depending on the traffic load and the network operator’s policy regarding energy reduction versus latency. Our results show that even in delay-sensitive scenarios, high energy gains can be achieved in low and moderate traffic loads, respectively 55% and 10%, without inducing an extra latency. Starting from a certain traffic load (approximately 30%), ASM should not be implemented in case of stringent latency constraint.