Most of recent methods for action/activity recognition, usually based on static classifiers, have achieved improvements by integrating context of local interest point features like Spatio-Temporal Interest Points by characterizing their neighborhood under different scales. In this paper, we propose a new approach that explicitly models the sequential aspect of activities. First a sliding window segmentation technique splits the video stream into overlapping short segments. Each window is characterized by a local Bag of Words of interest points encoded by motion information. A first-layer Support Vector Machine provides for each window a vector of conditional class probabilities that summarizes all discriminant information that is relevant for sequence recognition. The sequence of these stochastic vectors is then fed to a Hidden Conditional Random Field for inference at the sequence level. We also show how our approach can be naturally extended to the problem of conjoint segmentation and recognition of a sequence of action classes within a continuous video stream. We have tested our model on various human action and activity datasets and the obtained results compare favorably with current state of the art