Mixed-initiative cooperative control can help overcome the limitations of autonomous robots by introducing a human operator into the control loop. Developing new computational models for cooperative control presents several key questions : how to account for the different perceptual and action abilities of the robot versus the human operator, the different costs of their actions, and the fact that human operators cannot switch their attention and take over control instantly. We present a decision-theoretic approach to solve these problems using Mixed-Initiative Markov Decision Processes (MI-MDPs). This framework allows us to compute optimal plans that not only tell the robot what actions to perform, but also when to request human attention or transfer control to the human operator. The result is a model with a varying degree of autonomy, particularly suitable for robots exploring a domain with regions that are too complex or risky for autonomous operation. Experimental results confirm the effectiveness of this approach in common scenarios in which fully-autonomous operation or continuous human control are both undesirable.