Video-based recognition of activities of daily living (ADLs) is being used in ambient assisted living systems in order to support independent living of old people. In this work, we propose a new multimodal ADL recognition method by modeling the correlation between motion and object information. We encode motion using dense interest point trajectories which are robust to occlusion and speed variability. We formulate the learning problem using a two-layer SVM hidden conditional random field (HCRF) recognition model that is particularly relevant for multimodal sequence recognition. This hierarchical classifier opti-mally combines the discriminative power of SVM and the long-range feature dependencies modeling by the HCRF