Learning an effective similarity metric between the image representations is key to the success of the recent advances in visual search tasks (e.g. verification or zero shot learning). Although the metric learning part is well addressed, this metric is usually computed over the average of the extracted deep features. This representation is then trained to be discriminative, however, these deep features tend to be scattered across the feature space. That way, the representations are not robust for the feature outliers, object occlusions, background variations, etc. In this paper, we tackle this scattering problem with a distribution-aware regularization named HORDE. This regularizer is designed to ensure that the feature distributions from two dissimilar images are well separated in the feature space. In the case of similar images, the feature distributions are enforced to be localized in the same feature space. We prove theoretically that our regularizer upper and lower bounds wellknown distances between distributions, such as the Maximum Mean Discrepancy and the Wasserstein distance. Empirically, HORDE consistently improves deep metric learning architectures, leading to state-of-the-art results on 3 out of 4 deep metric learning datasets.