This paper introduces the information criteria in feature selection framework. Information criteria are integrated in feature selection scheme to select subset candidates. The accuracy of the proposed approach is based on the quality of the probability density approximations of these features. They are obtained using histograms optimized thanks to the adaptive arithmetic coding principles. Tests on simulated data and references are made. Multiple classifiers are used. The correct classification rate shows, the importance of this tools and its ability to select a best subsets. Generally, the subsets produce a good characterization of classes which the data belong.