In this work, we address the problem of tracking faces and facial actions in a single video sequence. The main contributions of the paper are as follows. First, we develop a particle filter based framework for tracking the global 3-D motion of a face using a statistical facial appearance model. Second, we propose a framework for tracking the 3-D face pose as well as the local motion of inner features of the face due for instance to spontaneous facial actions, using an adaptive appearance model. We allow the statistics of the facial appearance as well as the dynamics to be adaptively updated during tracking. Third, we propose a variant of the second framework based on a heuristic search. Tracking real video sequences demonstrated the effectiveness of the developed methods. Accurate tracking was obtained even in the presence of perturbing factors including significant head pose and facial expression variations, occlusions, and illumination changes