Discrete event modeling and simulation and reinforcement learning are two frameworks suited for cyberphysical system design, which, when combined, can give powerful tools for system optimization or decision making process for example. This paper describes how discrete event modeling and simulation could be integrated into reinforcement learning concepts and tools in order to assist in the realization of reinforcement learning systems, more specially considering the temporal, hierarchical, and multi-agent aspects. An overview of these different improvements are given based on the implementation of the Q-Learning reinforcement learning algorithm in the framework of the Discrete Event system Specification (DEVS) and System Entity Structure (SES) formalisms.