Spatio-temporal interest points (STIPs) have recently become a mainstream technique for encoding human action video sequences. These local features overcome the complex issues of background subtraction and human tracking, and encode relevant motion information for recognition. Nonetheless, STIPs may result not only from actions themselves but from other factors such as person identity. This paper addresses modeling the multi-modal aspect of motions using Multilinear Tensor Analysis, with the goal of detaching parameters that are relevant to action recognition from those that are irrelevant (related to person identity). This modeling of space time interest points by a tensor representation is a new contribution to human action analysis. Experiments carried out on the Weizmann and KTH datasets give the interesting finding that STIP-based features are not only related to action types but may also be sensitive to person identity or style.