This paper presents an alternative to the mainstream approach of STIP-based SVM recognition for human recognition. First, it studies whether or not whole silhouette representation by Histogram-of-Oriented-Gradients (HOG) or Histogram-of-Optical-Flow (HOF) descriptors is more discriminated when compared to sparse spatio-temporal interest points (STIPs). Second, it investigates whether explicitly modeling the temporal order of features using continuous HMMs outperforms the standard Bag-of-Words (BoW) representation that overlooks such an order. When both whole silhouette representation and temporal order modeling are combined, a significant improvement is shown on the Weizmann database over STIP-based SVM recognition.