In this paper, we propose a novel approach to capture the dynamic deformation caused by facial expressions. The proposed method is concentrated on the spatiotemporal plane which is not well explored. It uses the moments as features to describe the movements of essential components such as eyes and mouth on vertical time plane. The system we developed can automatically recognize the expression on images as well as on image sequences. The experiments are performed on 348 sequences from 95 subjects in Cohn-Kanade database and obtained good results as high as 96.1% in 7-class recognition for frames and 98.5% in 6-class for sequences.