We propose a reinforcement learning approach to learning to teach. Following Torrey and Taylor’s framework [18], an agent (the “teacher”) advises another one (the “student”) by suggesting actions the latter should take while learning a specific task in a sequential decision problem; the teacher is limited by a “budget” (the number of times such advice can be given). Our approach assumes a principled decision-theoretic setting; both the student and the teacher are modeled as reinforcement learning agents. We provide experimental results with the Mountain car domain, showing how our approach outperforms the heuristics proposed by Torrey and Taylor [18]. Moreover, we propose a technique for a student to take into account advice more efficiently and we experimentally show that performances are improved in Torrey and Taylor’s setting.