In this paper, we propose a method to track multiple deformable objects in sequences (with a static camera) in and beyond the visible spectrum by combining Gabor filtering and clustering. The idea is to sample moving areas between two frames by randomly positioning samples over high magnitude area of a motion likelihood function. These points are then clustered to obtain one class for each moving object. The novelty in our method is in using cluster information from the previous frame to classify new samples in the current frame: we call that a recursive clustering. This makes our method robust to occlusions, objects entering and leaving the field of view, objects stopping and starting, and moving objects getting really close to each other.