This article presents a clustering algorithm to determine the optimal number of components in a Gaussian mixture. The principle is to start from an important number of mixture components then group the multivariate normal distributions into clusters using the divergence, a weighted symmetric, distortion measure based on the Kullback-Leibler distance. The optimal cut in the tree, i.e. the clustering, satisfies criteria based on either the minimum amount of available training data or dissimilarities between clusters. The performance of this algorithm is compared favorably against a reference system and a likelihood loss based clustering system. The tree cutting criteria are also discussed. About an hour of Ester, a French broadcast news database is used for the recognition experiments. Performance is significantly improved and the word error rate decreases by about4.8%, where the confidence interval is 1%.