We propose a novel probabilistic tracking algorithm based on an elastic blob ensemble (EBE) which is applicable to track flexible objects, like human upper body composed of head, torso, and hips. It outputs a coarse motion cue in the form of the object's location and orientation together with the location of the blobs. The main assumption is that the orientation of the whole object does not change much between neighboring frames. Hence, a discrete solution space is created in the current frame around the blobs' positions from the previous frame. Our model then promotes solutions whose orientations are close to the prior orientation, which match the modeled to the observed appearance well, and which follow modeled spatial configuration. It combines the strengths of three popular approaches to visual tracking: mean-shift tracker, particle filtering, and pictorial structures. As a result, the proposed framework leads to a robust and fast tracker of a person at the rate of 15-30 fps on a regular desktop PC