Intrusion detection is an important topic in cybersecurity research, but the evaluation methodology has remained stagnant despite advancements including the use of machine learning. In this paper, we design a comprehensive evaluation framework for Machine Learning (ML)-based IDS and take into account the unique aspects of ML algorithms, their strengths, and weaknesses. The framework design is inspired by both i) traditional IDS evaluation methods and ii) recommendations for evaluating ML algorithms in diverse application areas. Data quality being the key to machine learning, we focus on datadriven evaluation by exploring data-related issues.