Action recognition in video and still image is one of the most challenging research topics in pattern recognition and computer vision. This paper proposes a new method for video action classification based on 3D Zernike moments. These last ones aim to capturing both structural and temporal information of a time varying sequence. The originality of this approach consists to represent actions in video sequences by a three-dimension shape obtained from different silhouettes in the space-time volume. In fact, the given video is segmented in space-time volume. Then, silhouettes are extracted from obtained images of the video sequences volumes and 3D Zernike moments are computed for video, based on silhouettes volumes. Finally, least square version of SVM (LSSVM) classifier with extracted features is used to classify actions in videos. To evaluate the proposed approach, it was applied on a benchmark human action dataset. The experimentations and evaluations show efficient results in terms of action characterizations and classification. Further more, it presents several advantages such as simplicity and respect of silhouette movement progress in the video guaranteed by 3D Zernike moment.