This paper proposes an original and uni ed toolbox to evaluate nancial crisis Early Warning Systems (EWS). It presents four main advantages. First, it is a model free method which can be used to asses the forecasts issued from di erent EWS (probit, logit, markov switching models, or combinations of models). Second, this toolbox can be applied to any type of crisis EWS (currency, banking, sovereign debt, etc.). Third, it does not only provide various criteria to evaluate the (absolute) validity of EWS forecasts but also proposes some tests to compare the relative performance of alternative EWS. Fourth, our toolbox can be used to evaluate both in-sample and out-of-sample forecasts. Applied to a logit model for twelve emerging countries we show that the yield spread is a key variable to predict currency crises exclusively for South-Asian countries. Besides, the optimal cut-o correctly allows us to identify now on average more than 2/3 of the crisis and calm periods.