This paper proposes a planning management framework dedicated to dynamic systems. It deals with intentional BDI agents in aim to maximize the satisfaction of their intentions taking into account the spatio-temporal context. The search for the optimal solution is based on an original and formal structure called Contextual Planning System (CPS). The CPS handles the satisfaction of concurrent plans associated with intentions. It looks for efficient guidance to improve the satisfaction of the agent's intentions with respect to its current context. Our paper goes on to show how to take advantage of context-aware information representing spatio-temporal data related to the past-experience of action plan executions. Our objective is to improve the CPS mechanism by the use of reinforcement learning strategy. A prototype tool associated with automatic generation of experiences validates our approach on a significant number of experiences.