Over the last fifty years, societies across the world have experienced multiple periods of energy insufficiency with the most recent one being the 2022 global energy crisis. In addition, the electric power industry has been experiencing a steady increase in electricity consumption since the second industrial revolution because of the widespread usage of electrical appliances and devices. Newer devices are equipped with sensors and actuators, they can collect a large amount of data that could help in power management. However, current energy management approaches are mostly applied to limited types of devices in specific domains and are difficult to implement in other scenarios. They fail when it comes to their level of autonomy, flexibility, and genericity. To address these shortcomings, we present, in this paper, an automated energy management approach for connected environments based on generating power estimation models, representing a formal description of energy-related knowledge, and using reinforcement learning (RL) techniques to accomplish energy-efficient actions. The architecture of this approach is based on three main components: power estimation models, knowledge base, and intelligence module. Furthermore, we develop algorithms that exploit knowledge from both the power estimator and the ontology, to generate the corresponding RL agent and environment. We also present different reward functions based on user preferences and power consumption. We illustrate our proposal in the smart home domain. An implementation of the approach is developed and two validation experiments are conducted. Both case studies are deployed in the context of smart homes: (a) a living room with a variety of devices and (b) a smart home with a heating system. The obtained results show that our approach performs well given the low convergence period, the high level of user preferences satisfaction, and the significant decrease in energy consumption.