In this paper, we propose a novel method to track an object whose appearance is evolving in time. The tracking procedure is performed by a particle filter algorithm in which all possible appearance models are explicitly considered using a mixture decomposition of the likelihood. Then, the component weights of this mixture are conditioned by both the state and the current observation. Moreover, the use of the current observation makes the estimation process more robust and allows handling complementary features, such as color and shape information. In the proposed approach, these estimated component weights are computed using a Support Vector Machine. Tests on a mouth tracking problem show that the multiple appearance model outperforms classical single appearance likelihood.