Due to the sensitive nature of diabetes-related data, preventing them from being easily shared between studies, and the wide discrepancies in their data processing pipeline, progress in the field of glucose prediction is hard to assess. To address this issue, we introduce GLYFE (GLYcemia Forecasting Evaluation), a benchmark of machine learning-based glucose predictive models. We present the accuracy and clinical acceptability of nine different models coming from the literature, from standard autoregressive to more complex neural network-based models. These results are obtained on two different datasets, namely UVA/Padova Type 1 Diabetes Metabolic Simulator (T1DMS) and Ohio Type-1 Diabetes Mellitus (OhioT1DM), featuring artificial and real type 1 diabetic patients respectively. By providing extensive details about the data flow as well as by providing the whole source code of the benchmarking process, we ensure the reproducibility of the results and the usability of the benchmark by the community. Those results serve as a basis of comparison for future studies. In a field where data are hard to obtain, and where the comparison of results from different studies is often irrelevant, GLYFE gives the opportunity of gathering researchers around a standardized common environment.