Boosting Diversity in Regression Ensembles - Archive ouverte HAL Access content directly
Journal Articles Statistical Analysis and Data Mining Year : 2023

Boosting Diversity in Regression Ensembles

Mathias Bourel
  • Function : Author
Yannig Goude
  • Function : Author
Jean‐michel Poggi
  • Function : Author


The practical interest of using ensemble methods has been highlighted in several works. Aggregating predictors leads very often to improve the performance of a single one. A fruitful recipe is to generate several predictors from a single one by perturbing the learning set and, instead of selecting the best one, to aggregate them. Bagging, boosting and Random forests are examples of such strategies useful both for classification and regression problems. A key ingredient to properly analyse the improvement of prediction performance is the diversity of the predictors ensemble. In the regression case, aggregation is mainly interested on how to generate individual predictors to improve quadratic prediction performance. We look for enhancing these methods by using the concept of diversity (also known as negative correlation learning). We propose an algorithm to enrich the set of original individual predictors using a gradient boosting-based method by incorporating a diversity term to guide the gradient boosting iterations. The idea is to progressively generate predictors by boosting diversity, this modification induces some kind of suboptimality of the individual learners but improve the ensemble. Then, we establish a convergence result ensuring that the associated optimisation strategy converges to a global optimum. Finally, we show by means of numerical experiments the appropriateness of our procedure and examine not only the final predictor or the aggregated one but also the generated sequence. First, on a simulated dataset, we illustrate and study the method with respect to the family of predictors as well the parameters to be tuned (diversity weight and gradient step). Second, real-world electricity demand datasets are considered opening the application of such ideas to the forecasting context.
Fichier principal
Vignette du fichier
main_modif-jc.pdf (506.66 Ko) Télécharger le fichier
Boosting (468.88 Ko) Télécharger le fichier
Origin Files produced by the author(s)
Origin Files produced by the author(s)

Dates and versions

hal-03041309 , version 1 (04-12-2020)



Mathias Bourel, Jairo Cugliari, Yannig Goude, Jean‐michel Poggi. Boosting Diversity in Regression Ensembles. Statistical Analysis and Data Mining, 2023, 17 (1), ⟨10.1002/sam.11654⟩. ⟨hal-03041309⟩
291 View
287 Download



Gmail Mastodon Facebook X LinkedIn More