Analyzing the Impact of Undersampling on the Benchmarking and Configuration of Evolutionary Algorithms - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Analyzing the Impact of Undersampling on the Benchmarking and Configuration of Evolutionary Algorithms

Résumé

The stochastic nature of iterative optimization heuristics leads to inherently noisy performance measurements. Since these measurements are often gathered once and then used repeatedly, the number of collected samples will have a significant impact on the reliability of algorithm comparisons. We show that care should be taken when making decisions based on limited data. Particularly, we show that the number of runs used in many benchmarking studies, e.g., the default value of 15 suggested by the COCO environment, can be insufficient to reliably rank algorithms on well-known numerical optimization benchmarks. Additionally, methods for automated algorithm configuration are sensitive to insufficient sample sizes. This may result in the configurator choosing a "lucky" but poor-performing configuration despite exploring better ones. We show that relying on mean performance values, as many configurators do, can require a large number of runs to provide accurate comparisons between the considered configurations. Common statistical tests can greatly improve the situation in most cases but not always. We show examples of performance losses of more than 20%, even when using statistical races to dynamically adjust the number of runs, as done by irace. Our results underline the importance of appropriately considering the statistical distribution of performance values.
Fichier principal
Vignette du fichier
GECCO-variability-2204.09353.pdf (7.36 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03718886 , version 1 (13-07-2022)

Identifiants

Citer

Diederick Vermetten, Hao Wang, Manuel López-Ibañez, Carola Doerr, Thomas Bäck. Analyzing the Impact of Undersampling on the Benchmarking and Configuration of Evolutionary Algorithms. GECCO '22: Genetic and Evolutionary Computation Conference, Jul 2022, Boston, United States. ⟨10.1145/3512290.3528799⟩. ⟨hal-03718886⟩
24 Consultations
14 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More