The security of elderly people living alone is a major issue. A system that detects anomalies can be useful for both individual and retirement homes. In this paper, we present an adaptive human tracking method built on particle filter, using depth and thermal information based on the velocity and the position of the head. The main contribution of this paper is the fusion of information to improve tracking. For each frame, there is a new combination of coefficients for each particle based on an adaptive weighting. Results show that the tracking method can deal with the cases of fast motion (fall), partial occultation and scale variation. To assess the impact of fusion on the tracking process, the robustness and accuracy of the method are tested on a variety of challenging scenarios with or without depth-thermal fusion.