In this paper, we propose a mean-shift formulation allowing spatiotemporal clustering of video streams, and possibly extensible to other multivariate evolving data. Our formulation enables causal or omniscient filtering of spatiotemporal data, which is robust to total object occlusions. It embeds a new clustering algorithm within the filtering procedure that will group samples and reduce their number over the iterations. Based on our formulation, we express similar approaches and assess their robustness on real video sequences.