Speech emotion is high semantic information and its automatic analysis may have many applications such as smart human-computer interactions or multimedia indexing. As a pattern recognition problem, the feature selection and the structure of the classifier are two important aspects for automatic speech emotion classification. In this paper, we propose a novel feature selection scheme based on the evidence theory. Furthermore, we also present a new automatic approach for constructing a hierarchical classifier, which allows better performance than a global classifier as it is mostly used in the literature. Experimented on the Berlin database, our approach showed its effectiveness, scoring a recognition rate up to 78.64%.