A new method for action recognition is proposed by revisit-ing LBP-based dynamic texture operators. It captures the similarity of motion around keypoints tracked by a realtime semi-dense point track-ing method. The use of self-similarity operator allows to highlight the geometric shape of rigid parts of foreground object in a video sequence. Inheriting from the efficient representation of LBP-based methods and the appearance invariance of patch matching method, the method is well designed for capturing action primitives in unconstrained videos. Action recognition experiments, made on several academic action datasets vali-date the interest of our approach.