In this paper, we investigate network sleep mode schemes for reducing energy consumption of radio access networks. We first propose, using Markov Decision Processes (MDPs), an optimal controller that associates to each traffic an activation/deactivation policy that maximizes a multiple objective function of the Quality of Service (QoS) and the energy consumption. We focus on a practical implementation issue, namely ping-pong effect resulting in unnecessary ON/OFF oscillations, that may affect the stability of the system. We illustrate our results numerically using theoretical models of the radio access network, and apply the developed mechanisms on a large-scale network simulator. Knowing that an offline optimization is not suitable for a large-scale network nor does it fit all traffic configurations, we propose, using an online controller that derives dynamically the optimal policy based on the dynamics of users in the cell. The design of our online controller is based on a simple ∊-greedy algorithm and learns the optimal threshold policy for activation/deactivation of network resources.