This paper shows the information criteria (IC) performances in feature selection framework. Feature selection aims to select a representative subset among a wide set of features. We apply this approach to classify an hand segmented image. The performance is tested using various feature selection schemes (SFS, SBS, SFFS and SBFS) to select the candidate subsets. The accuracy of the approach is based on a good quality of the joint probability density approximation of the combined features. They are obtained using histogram optimized thanks to the adaptive arithmetic coding principles. Our approach is tested on different reference data. The subsets quality is evaluated using correct classification rate computed on multiple classifiers. Results show stability and convergence properties of this tool and its ability to select representative subsets (in the sense that the subset of feature is a good characterization of the classes in which the data belong). Information Criteria could be used for feature selection as a good alternative to other criteria.