Taking into Account the Specificities of Space Weather for the Evaluation of Machine-Learning-Based Forecasts
Résumé
Over the last decade we have witnessed a rapid growth in the use of Machine Learning and Deep Learning methods to solve Space Weather problems. In particular, the number of studies on the application of these methods to forecasting problems (whether of geomagnetic indices, particle fluxes in the radiation belts or other physical parameters) has increased significantly. However, the comparison of these different studies is often made difficult because the data used or the evaluation metrics employed are not always exactly the same. Furthermore, the use of data-driven methods implies additional constraints to ensure that the results are not biased. In this presentation we will address these different difficulties by focusing on three main points. First, we will discuss the use of metrics dedicated to the measurement of time lags between two time series, in addition to classical metrics such as the mean-squared error. We will also talk about good practices to adopt when possible, such as cross-validation, in order not to bias our results, and to make them as representative as possible, which is not immediate when using a limited data set. Finally, we will argue for the establishment of minimum evaluation benchmarks for the prediction of certain parameters (such as the geomagnetic index Kp), following the example of the meteorological and climatological community.