Advanced Sleep Modes (ASMs) correspond to a gradual deactivation of the Base Station (BS)'s components in order to reduce its Energy Consumption (EC). Different levels of Sleep Modes (SMs) can be considered according to the transition time (deactivation and activation durations) of each component. We propose in this paper a management solution for ASMs based on Q-learning approach. The target is to find the optimal durations for each SM level according to the requirements of the network operator in terms of EC reduction and delay constraints. The proposed solution shows that even with a high constraint on the delay, we can achieve high energy savings (almost 57% of EC reduction) without inducing any impact on the delay. When the delay constraint is relaxed, we can achieve up to almost 90% of energy savings