Decarbonizing the grid is recognized worldwide as one of the objectives for the next decades. Its success depends on our ability to massively deploy renewable resources, but to fully benefit from those, grid flexibility is needed. In this paper we put forward the design of a benchmark that will allow for the systematic measurement of demand response programs' effectiveness, information that we do not currently have. Furthermore, we explain how the proposed benchmark will facilitate the use of Machine Learning techniques in grid flexibility applications.