Locating human actions in videos is challenging because ofthe complexity and variability of human motions, as well asof the amount of video data to be searched. Different actionscan be composed of similar short motions and only differ bytheir temporal ordering or relative durations. We propose amethod that detects and locates a set of actions in a videodatabase by taking into account their temporal structure at theframe level. While other methods aggregate frames into ac-tion parts, we leverage the complementarity between aggre-gation and frame level comparison of sequences. With theaim to address large scale retrieval, we introduce a two-levelcascade. The first level employs inexpensive aggregation tofilter out a large part of the video. The second level of thecascade applies to the remaining sequences more discrimi-nant comparisons using frame alignment. Evaluations on the?Smoking and Drinking? and ?MSR Action II? datasets showstate of the art results, as well as efficient detection and lowstorage requirements.