Extreme multi-class classification concerns classification problems with very large number of classes, up to several millions. Such problems have now become quite frequent in many practical applications. Until recently, most classification methods had inference complexity at least linear in the number of classes. Several directions have been recently explored for limiting this complexity, but the challenge of learning an optimal compromise between inference complexity and classification accuracy is still largely open. We propose here a novel ensemble learning approach, where classifiers are dynamically chosen among a pre-trained set of classifiers and are iteratively combined in order to achieve an efficient trade-off between inference complexity and classification accuracy. The proposed model uses statistical bounds to discard during the inference process irrelevant classes and to choose the most informative classifier with respect to the information gathered during the previous steps. Experiments on real datasets of recent challenges show that the proposed approach is able to achieve a very high classification accuracy in comparison to baselines and recent proposed approaches for similar inference time complexity.