This article deals with facial segmentation and liptracking with feedback control for real-time animation of a synthetic 3D face model. Straightforward approaches consist in two successive steps: video analysis then synthesis. Our approach departs from the previous ones in that we build a global analysis/synthesis processing loop, where the image analysis needs the 3D synthesis and conversely. A first facial segmentation is computed according to which the 3D face model is positionned. Then the feedback loop, implemented from the 3D animated model back to the input pixel segmentation algorithm, helps to correct some (few) control points that were badly tracked, which are detected by measuring the vertical distance between lip contour and corresponding 3D face model. When this distance is too big, we re-enter into the image segmentation process and zoom-in inside a few regions of interest (ROI) where the algorithm is run again, with a new set of tuning parameters better suited to the pixel neighborhood context. In that way, the face segmentation is refined in order to extract more precise parameters. This approach is inspired from control theory with closed-loop systems. The contribution of the paper is to use simple image processing techniques, but to improve segmentation through the feedback loop. Results show that real-time and robust performances are achievable under real-world conditions, which are two key issues for face and lip tracking applications.