In this paper, we present a unified framework for the analysis of video databases by using Markov spatio-temporal random walks on graph. The proposed framework provides an efficient approach for clustering, data organization, dimension reduction and recognition. The aim of our work is to develop a vision-based approach for human behaviour recognition. Our contribution lies in three aspects. First, we employ 3D Zernike moments to encode the object of interest in a video clip. Then, we propose a new method to represent the video database as a weighted undirected graph where each vertex is a video clip. The weight of an edge between two video clips is defined by a Gaussian kernel on their 3D Zernike moments and their respective neighbourhoods in the feature space. Our objective is to obtain a robust low-dimensional space through spectral graph embedding which provides efficient keypoints transcription into an euclidean manifold, and allows to achieve higher classification accuracy through agglomerative categorization. Finally, we describe a variational framework for manifold denoising based on p-Laplacian, thereby lessening the negative impact of outliers, enhancing keypoints classification and thus, boosting the recognition accuracy. The proposed method is tested on the Weizmann and KTH human action datasets and on a hand gesture dataset. The retrieved results using the 3D Zernike moments prove that the proposed method can effectively capture the form of the behaviours with low order moments. Moreover, our framework allows to classify various behaviours and achieves a significant recognition rate.