Facial expression’s machine analysis is one of the most challenging problems in Human–Computer Interaction (HCI). Naturally, facial expressions depend on subtle movements of facial muscles to show emotional states. After having studied the relations between basic expressions and corresponding facial deformation models, we propose two new textons, VTB and moments on spatiotemporal plane, to describe the transformation of human face during facial expressions. These descriptors aim at catching both general shape changes and motion texture details. Therefore, modelling the temporal behaviour of facial expression captures the dynamic deformation of facial components. Finally, SVM based system is used to efficiently recognize the expression for a single image in sequence. Then, the probabilities of all the frames are used to predict the class of the current sequence. The experimental results are evaluated on both Cohan–Kanade and MMI databases. By comparison to other methods, the effectiveness of our method is clearly demonstrated