Intrusion detection systems are an important domain in cybersecurity research. Countless solutions have been proposed, continuously improving upon one another. Yet, and despite the introduction of distinct approaches, including machine-learning methods, the evaluation methodology has barely evolved. In this paper, we design a comprehensive evaluation framework for Machine Learning (ML)-based intrusion detection systems (IDS) and take into account the unique aspects of ML algorithms, their strengths and weaknesses. The framework design is inspired by both i) traditional IDS evaluation methods and ii) recommendations for evaluating ML algorithms in diverse application areas. Data quality being the key to machine learning, we focus on data-driven evaluation by exploring data-related issues. Our approach goes beyond evaluating intrusion detection performance (also known as effectiveness) and aims at proposing standard data manipulation methods to tackle robustness and stability. Finally, we evaluate our framework through a qualitative comparison with other IDS evaluation approaches from the state of the art.