We present a framework for controlling physics-based bipeds in a simulated environment, based on a variety of reference motions. Unlike existing methods for control based on reference motions, our framework does not require preprocessing of the reference motion, nor does it rely on inverse dynamics or on-line optimization methods for torque computation. It consists of three components: Proportional-Derivative Control to mimic motion characteristics, a specific form of Jacobian Transpose Control for balance control, and Covariance Matrix Adaption for off-line parameter optimization, based on a novel high-level reward function. The framework can easily be implemented using common off-the-shelf physics engines, and generates simulations at approximately 4× realtime on a single core of a modern PC. Our framework advances the state-of-the-art by demonstrating motions of a diversity and dynamic nature previously unseen in comparable methods, including squatting, bowing, kicking, and dancing motions. We also demonstrate its ability to withstand external perturbations and adapt to changes in character morphology.