We address the problem of multitarget region tracking within image sequences. Following recent work on joint segmentation and tracking as well as non-parametric modeling of color statistics, we develop an energy-minimization based approach using color histograms measures. As in a few other existing approaches, a single color probability distribution per object and background is handled. In this context, global histograms may be problematic for tracking in real scenes with cluttered backgrounds, where statistical color data is highly scattered, preventing the estimation of reliable color statistics for object/background discrimination. To overcome this limitation, we introduce a short-sight perception modeling of background, which concentrates on the vicinity of tracked objects and thus extract more consistent statistical data for accurate separation between objects and background. To account for temporal consistency, our energy is also endowed with a novel data term explicitly based on temporal variation of color distribution within objects and local background regions.