Detection of anomalies (i.e., outliers) in multi-dimensional data is a well-studied subject in machine learning. Unfortunately, unsupervised detectors provide no explanation about why a data point was considered as abnormal or which of its features (i.e. subspaces) exhibit at best its outlyingness. Such outlier explanations are crucial to diagnose the root cause of data anomalies and enable corrective actions to prevent or remedy their effect in downstream data processing. In this work, we present a comprehensive framework for comparing different unsupervised outlier explanation algorithms that are domain and detector-agnostic. Using real and synthetic datasets, we assess the effectiveness and efficiency of two point explanation algorithms (Beam [28] and RefOut [18]) ranking subspaces that best explain the outlyingness of individual data points and two explanation summarization algorithms (LookOut [15] and HiCS [17]) ranking subspaces that best exhibit as many outlier points from inliers as possible. To the best of our knowledge, this is the first detailed evaluation of existing explanation algorithms aiming to uncover several missing insights from the literature such as: (a) Is it effective to combine any explanation algorithm with any off-the-shelf outlier detector? (b) How is the behavior of an outlier detection and explanation pipeline affected by the number or the correlation of features in a dataset? and (c) What is the quality of summaries in the presence of outliers explained by subspaces of different dimensionality? * Work was done while the author was working at SAP. † This work received funding by the CY Initiative of Excellence (grant "Investissements d'Avenir" ANR-16-IDEX-0008) and developed during the author stay at the CY Advanced Studies, whose support is gratefully acknowledged.