In this paper, we aim to optimize the sequencing of learning activities using the Q-learning, a reinforcement learning method. The Q-learning agent decides each time which activity to propose to the student. The sequencing policy we propose is guided by the aim to improve efficiently the student knowledge state. Thus, the Q-learning learns a mapping of the student knowledge states to the optimal activity to perform in that state. In this paper, we tackle two main issues in implementing the Q-learning off-policy: the combinatorial explosion of the student knowledge states and the definition of the reward function allowing to improve efficiently the student knowledge state. We rely on the student model and the domain model to answer these two challenges. We carried out a study to evaluate the approach we propose on simulated students. We show that our approach is more efficient since it achieves better learning gain with fewer activities than a random policy or an expert based policy.