In this paper, we have proposed a novel framework to enable hierarchical image classification via statistical learning. By integrating the concept hierarchy for semantic image concept organization, a hierarchical mixture model is proposed to enable multi-level modeling of semantic image concepts and hierarchical classifier combination. Thus, learning the classifiers for the semantic image concepts at the high level of the concept hierarchy can be effectively achieved by detecting the presences of the relevant base-level atomic image concepts. To effectively learn the base-level classifiers for the atomic image concepts at the first level of the concept hierarchy, we have proposed a novel adaptive EM algorithm to achieve more effective model selection and parameter estimation. In addition, a novel penalty term is proposed to effectively eliminate the misleading effects of the outlying unlabeled images on semi-supervised classifier training. Our experimental results in a specific image domain of outdoor photos are very attractive.