Advanced Sleep Modes (ASM) correspond to a gradual deactivation of the base station's components according to the time needed by each of them to shut down then reactivate again. Each level of sleep has a different power consumption and imposes an extra delay on arriving traffic as it has to wait for the components to wake up and serve it. We present in this work a scalable management strategy of this feature based on Markov Decision Processes (MDP) in order to derive the optimal policy allowing to choose the best sleep level according to the traffic load and to the tradeoff between delay and energy consumption while ensuring a low complexity. Our results show that this solution is very promising and allows to achieve high energy saving (up to 91%) if there is no constraint on the delay, but even with a high constraint, the energy reduction can reach up to 52% while the impact on the delay is negligible.