In the context of web-scale taxonomies such as Directory Mozilla(www.dmoz.org), previous works have shown the existence of power law distribution in the size of the categories for every level in the taxonomy. In this work, we analyse how such high-level semantics can be leveraged to evaluate accuracy of hierarchical classifiers which automati- cally assign the unseen documents to leaf-level categories. The proposed method offers computational advantages over k-fold cross-validation.