There is extensive evidence that AI-based educational technology can effectively provide personalized support to students’ problem solving in many domains. In contrast, there has been limited work on AI-based environments that support exploratory activities, such as learning from interactive simulations or playing educational games. These activities are becoming increasingly widespread, especially in MOOCs (Massive Open On-Line Courses) and in other forms of self-directed learning, because they can foster motivation and grounded skills acquisition. However, not all learners possess the reasoning and study skills to fully benefit from these activities, calling for environments that can provide adaptive support for open-ended exploratory learning. Building such tools is challenging because it requires student models that assess behaviors for which there is no clear definition of correctness, as well as skills beyond those traditionally addressed by student models for problem solving. This chapter will review current advances of student modeling research in this direction.