Predictive performance of multi-model ensemble forecasts of COVID-19 across European nations
Katharine Sherratt
,
Hugo Gruson
(1)
,
Rok Grah
,
Helen Johnson
,
Rene Niehus
,
Bastian Prasse
,
Frank Sandmann
,
Jannik Deuschel
,
Daniel Wolffram
,
Sam Abbott
,
Alexander Ullrich
,
Graham Gibson
,
Evan Ray
,
Nicholas Reich
,
Daniel Sheldon
,
Yijin Wang
,
Nutcha Wattanachit
,
Lijing Wang
,
Jan Trnka
,
Guillaume Obozinski
,
Tao Sun
,
Dorina Thanou
,
Loic Pottier
,
Ekaterina Krymova
,
Jan Meinke
,
Maria Vittoria Barbarossa
,
Neele Leithauser
,
Jan Mohring
,
Johanna Schneider
,
Jaroslaw Wlazlo
,
Jan Fuhrmann
,
Berit Lange
,
Isti Rodiah
,
Prasith Baccam
,
Heidi Gurung
,
Steven Stage
,
Bradley Suchoski
,
Jozef Budzinski
,
Robert Walraven
,
Inmaculada Villanueva
,
Vit Tucek
,
Martin Smid
,
Milan Zajicek
,
Cesar Perez Alvarez
,
Borja Reina
,
Nikos Bosse
,
Sophie Meakin
,
Lauren Castro
,
Geoffrey Fairchild
,
Isaac Michaud
,
Dave Osthus
,
Pierfrancesco Alaimo Di Loro
,
Antonello Maruotti
,
Veronika Eclerova
,
Andrea Kraus
,
David Kraus
,
Lenka Pribylova
,
Bertsimas Dimitris
,
Michael Lingzhi Li
,
Soni Saksham
,
Jonas Dehning
,
Sebastian Mohr
,
Viola Priesemann
,
Grzegorz Redlarski
,
Benjamin Bejar
,
Giovanni Ardenghi
,
Nicola Parolini
,
Giovanni Ziarelli
,
Wolfgang Bock
,
Stefan Heyder
,
Thomas Hotz
,
David Singh
,
Miguel Guzman-Merino
,
Jose Aznarte
,
David Morina
,
Sergio Alonso
,
Enric Alvarez
,
Daniel Lopez
,
Clara Prats
,
Jan Pablo Burgard
,
Arne Rodloff
,
Tom Zimmermann
,
Alexander Kuhlmann
,
Janez Zibert
,
Fulvia Pennoni
,
Fabio Divino
,
Marti Catala
,
Gianfranco Lovison
,
Paolo Giudici
,
Barbara Tarantino
,
Francesco Bartolucci
,
Giovanna Jona Lasinio
,
Marco Mingione
,
Alessio Farcomeni
,
Ajitesh Srivastava
,
Pablo Montero-Manso
,
Aniruddha Adiga
,
Benjamin Hurt
,
Bryan Lewis
,
Madhav Marathe
,
Przemyslaw Porebski
,
Srinivasan Venkatramanan
,
Rafal Bartczuk
,
Filip Dreger
,
Anna Gambin
,
Krzysztof Gogolewski
,
Magdalena Gruziel-Slomka
,
Bartosz Krupa
,
Antoni Moszyński
,
Karol Niedzielewski
,
Jedrzej Nowosielski
,
Maciej Radwan
,
Franciszek Rakowski
,
Marcin Semeniuk
,
Ewa Szczurek
,
Jakub Zielinski
,
Jan Kisielewski
,
Barbara Pabjan
,
Kirsten Holger
,
Yuri Kheifetz
,
Markus Scholz
,
Biecek Przemyslaw
,
Marcin Bodych
,
Maciej Filinski
,
Radoslaw Idzikowski
,
Tyll Krueger
,
Tomasz Ozanski
,
Johannes Bracher
,
Sebastian Funk
Katharine Sherratt
- Function : Author
- PersonId : 1289051
- ORCID : 0000-0003-2049-3423
Hugo Gruson
- Function : Author
- PersonId : 171003
- IdHAL : hugo-gruson
- ORCID : 0000-0002-4094-1476
Rok Grah
- Function : Author
Helen Johnson
- Function : Author
Rene Niehus
- Function : Author
Bastian Prasse
- Function : Author
Frank Sandmann
- Function : Author
Jannik Deuschel
- Function : Author
Daniel Wolffram
- Function : Author
- PersonId : 1289052
- ORCID : 0000-0003-0318-3669
Sam Abbott
- Function : Author
Alexander Ullrich
- Function : Author
Graham Gibson
- Function : Author
Evan Ray
- Function : Author
Nicholas Reich
- Function : Author
Daniel Sheldon
- Function : Author
Yijin Wang
- Function : Author
- PersonId : 1289053
- ORCID : 0000-0003-4438-6366
Nutcha Wattanachit
- Function : Author
Lijing Wang
- Function : Author
Jan Trnka
- Function : Author
- PersonId : 1289054
- ORCID : 0000-0002-1786-7562
Guillaume Obozinski
- Function : Author
Tao Sun
- Function : Author
- PersonId : 1289055
- ORCID : 0000-0001-6357-6726
Dorina Thanou
- Function : Author
Loic Pottier
- Function : Author
Ekaterina Krymova
- Function : Author
Jan Meinke
- Function : Author
Maria Vittoria Barbarossa
- Function : Author
Neele Leithauser
- Function : Author
Jan Mohring
- Function : Author
Johanna Schneider
- Function : Author
- PersonId : 1289056
- ORCID : 0000-0002-9330-2838
Jaroslaw Wlazlo
- Function : Author
Jan Fuhrmann
- Function : Author
- PersonId : 1289057
- ORCID : 0000-0002-7091-3740
Berit Lange
- Function : Author
Isti Rodiah
- Function : Author
Prasith Baccam
- Function : Author
Heidi Gurung
- Function : Author
Steven Stage
- Function : Author
Bradley Suchoski
- Function : Author
Jozef Budzinski
- Function : Author
Robert Walraven
- Function : Author
Inmaculada Villanueva
- Function : Author
- PersonId : 1289058
- ORCID : 0000-0003-4940-085X
Vit Tucek
- Function : Author
Martin Smid
- Function : Author
Milan Zajicek
- Function : Author
- PersonId : 1289059
- ORCID : 0000-0002-3226-7266
Cesar Perez Alvarez
- Function : Author
Borja Reina
- Function : Author
Nikos Bosse
- Function : Author
Sophie Meakin
- Function : Author
Lauren Castro
- Function : Author
Geoffrey Fairchild
- Function : Author
Isaac Michaud
- Function : Author
Dave Osthus
- Function : Author
Pierfrancesco Alaimo Di Loro
- Function : Author
Antonello Maruotti
- Function : Author
- PersonId : 1289060
- ORCID : 0000-0001-8377-9950
Veronika Eclerova
- Function : Author
- PersonId : 1289061
- ORCID : 0000-0001-8476-7740
Andrea Kraus
- Function : Author
David Kraus
- Function : Author
Lenka Pribylova
- Function : Author
Bertsimas Dimitris
- Function : Author
Michael Lingzhi Li
- Function : Author
Soni Saksham
- Function : Author
Jonas Dehning
- Function : Author
Sebastian Mohr
- Function : Author
Viola Priesemann
- Function : Author
- PersonId : 1289062
- ORCID : 0000-0001-8905-5873
Grzegorz Redlarski
- Function : Author
Benjamin Bejar
- Function : Author
Giovanni Ardenghi
- Function : Author
Nicola Parolini
- Function : Author
Giovanni Ziarelli
- Function : Author
Wolfgang Bock
- Function : Author
Stefan Heyder
- Function : Author
- PersonId : 1086603
- ORCID : 0000-0002-3371-4072
Thomas Hotz
- Function : Author
David Singh
- Function : Author
Miguel Guzman-Merino
- Function : Author
Jose Aznarte
- Function : Author
David Morina
- Function : Author
Sergio Alonso
- Function : Author
- PersonId : 1289063
- ORCID : 0000-0002-3989-8757
Enric Alvarez
- Function : Author
Daniel Lopez
- Function : Author
Clara Prats
- Function : Author
- PersonId : 1289064
- ORCID : 0000-0002-1398-7559
Jan Pablo Burgard
- Function : Author
- PersonId : 1289065
- ORCID : 0000-0002-5771-6179
Arne Rodloff
- Function : Author
Tom Zimmermann
- Function : Author
Alexander Kuhlmann
- Function : Author
Janez Zibert
- Function : Author
Fulvia Pennoni
- Function : Author
Fabio Divino
- Function : Author
Marti Catala
- Function : Author
Gianfranco Lovison
- Function : Author
Paolo Giudici
- Function : Author
Barbara Tarantino
- Function : Author
Francesco Bartolucci
- Function : Author
Giovanna Jona Lasinio
- Function : Author
Marco Mingione
- Function : Author
Alessio Farcomeni
- Function : Author
- PersonId : 1289066
- ORCID : 0000-0002-7104-5826
Ajitesh Srivastava
- Function : Author
Pablo Montero-Manso
- Function : Author
Aniruddha Adiga
- Function : Author
Benjamin Hurt
- Function : Author
Bryan Lewis
- Function : Author
- PersonId : 1289067
- ORCID : 0000-0003-0793-6082
Madhav Marathe
- Function : Author
Przemyslaw Porebski
- Function : Author
- PersonId : 1289068
- ORCID : 0000-0001-8012-5791
Srinivasan Venkatramanan
- Function : Author
Rafal Bartczuk
- Function : Author
- PersonId : 1289069
- ORCID : 0000-0002-0433-7327
Filip Dreger
- Function : Author
Anna Gambin
- Function : Author
Krzysztof Gogolewski
- Function : Author
- PersonId : 1289070
- ORCID : 0000-0001-5523-5198
Magdalena Gruziel-Slomka
- Function : Author
Bartosz Krupa
- Function : Author
Antoni Moszyński
- Function : Author
Karol Niedzielewski
- Function : Author
Jedrzej Nowosielski
- Function : Author
Maciej Radwan
- Function : Author
Franciszek Rakowski
- Function : Author
Marcin Semeniuk
- Function : Author
Ewa Szczurek
- Function : Author
Jakub Zielinski
- Function : Author
- PersonId : 1289071
- ORCID : 0000-0001-8935-8137
Jan Kisielewski
- Function : Author
Barbara Pabjan
- Function : Author
Kirsten Holger
- Function : Author
Yuri Kheifetz
- Function : Author
Markus Scholz
- Function : Author
Biecek Przemyslaw
- Function : Author
Marcin Bodych
- Function : Author
Maciej Filinski
- Function : Author
Radoslaw Idzikowski
- Function : Author
Tyll Krueger
- Function : Author
Tomasz Ozanski
- Function : Author
Johannes Bracher
- Function : Author
Sebastian Funk
- Function : Author
- PersonId : 778295
- ORCID : 0000-0002-2842-3406
Abstract
Background: Short-term forecasts of infectious disease burden can contribute to situational awareness and aid capacity planning. Based on best practice in other fields and recent insights in infectious disease epidemiology, one can maximise the predictive performance of such forecasts if multiple models are combined into an ensemble. Here, we report on the performance of ensembles in predicting COVID-19 cases and deaths across Europe between 08 March 2021 and 07 March 2022. Methods: We used open-source tools to develop a public European COVID-19 Forecast Hub. We invited groups globally to contribute weekly forecasts for COVID-19 cases and deaths reported by a standardised source for 32 countries over the next 1–4 weeks. Teams submitted forecasts from March 2021 using standardised quantiles of the predictive distribution. Each week we created an ensemble forecast, where each predictive quantile was calculated as the equally-weighted average (initially the mean and then from 26th July the median) of all individual models’ predictive quantiles. We measured the performance of each model using the relative Weighted Interval Score (WIS), comparing models’ forecast accuracy relative to all other models. We retrospectively explored alternative methods for ensemble forecasts, including weighted averages based on models’ past predictive performance. Results: Over 52 weeks, we collected forecasts from 48 unique models. We evaluated 29 models’ forecast scores in comparison to the ensemble model. We found a weekly ensemble had a consistently strong performance across countries over time. Across all horizons and locations, the ensemble performed better on relative WIS than 83% of participating models’ forecasts of incident cases (with a total N=886 predictions from 23 unique models), and 91% of participating models’ forecasts of deaths (N=763 predictions from 20 models). Across a 1–4 week time horizon, ensemble performance declined with longer forecast periods when forecasting cases, but remained stable over 4 weeks for incident death forecasts. In every forecast across 32 countries, the ensemble outperformed most contributing models when forecasting either cases or deaths, frequently outperforming all of its individual component models. Among several choices of ensemble methods we found that the most influential and best choice was to use a median average of models instead of using the mean, regardless of methods of weighting component forecast models. Conclusions: Our results support the use of combining forecasts from individual models into an ensemble in order to improve predictive performance across epidemiological targets and populations during infectious disease epidemics. Our findings further suggest that median ensemble methods yield better predictive performance more than ones based on means. Our findings also highlight that forecast consumers should place more weight on incident death forecasts than incident case forecasts at forecast horizons greater than 2 weeks. Funding: AA, BH, BL, LWa, MMa, PP, SV funded by National Institutes of Health (NIH) Grant 1R01GM109718, NSF BIG DATA Grant IIS-1633028, NSF Grant No.: OAC-1916805, NSF Expeditions in Computing Grant CCF-1918656, CCF-1917819, NSF RAPID CNS-2028004, NSF RAPID OAC-2027541, US Centers for Disease Control and Prevention 75D30119C05935, a grant from Google, University of Virginia Strategic Investment Fund award number SIF160, Defense Threat Reduction Agency (DTRA) under Contract No. HDTRA1-19-D-0007, and respectively Virginia Dept of Health Grant VDH-21-501-0141, VDH-21-501-0143, VDH-21-501-0147, VDH-21-501-0145, VDH-21-501-0146, VDH-21-501-0142, VDH-21-501-0148. AF, AMa, GL funded by SMIGE - Modelli statistici inferenziali per governare l'epidemia, FISR 2020-Covid-19 I Fase, FISR2020IP-00156, Codice Progetto: PRJ-0695. AM, BK, FD, FR, JK, JN, JZ, KN, MG, MR, MS, RB funded by Ministry of Science and Higher Education of Poland with grant 28/WFSN/2021 to the University of Warsaw. BRe, CPe, JLAz funded by Ministerio de Sanidad/ISCIII. BT, PG funded by PERISCOPE European H2020 project, contract number 101016233. CP, DL, EA, MC, SA funded by European Commission - Directorate-General for Communications Networks, Content and Technology through the contract LC-01485746, and Ministerio de Ciencia, Innovacion y Universidades and FEDER, with the project PGC2018-095456-B-I00. DE., MGu funded by Spanish Ministry of Health / REACT-UE (FEDER). DO, GF, IMi, LC funded by Laboratory Directed Research and Development program of Los Alamos National Laboratory (LANL) under project number 20200700ER. DS, ELR, GG, NGR, NW, YW funded by National Institutes of General Medical Sciences (R35GM119582; the content is solely the responsibility of the authors and does not necessarily represent the official views of NIGMS or the National Institutes of Health). FB, FP funded by InPresa, Lombardy Region, Italy. HG, KS funded by European Centre for Disease Prevention and Control. IV funded by Agencia de Qualitat i Avaluacio Sanitaries de Catalunya (AQuAS) through contract 2021-021OE. JDe, SMo, VP funded by Netzwerk Universitatsmedizin (NUM) project egePan (01KX2021). JPB, SH, TH funded by Federal Ministry of Education and Research (BMBF; grant 05M18SIA). KH, MSc, YKh funded by Project SaxoCOV, funded by the German Free State of Saxony. Presentation of data, model results and simulations also funded by the NFDI4Health Task Force COVID-19 ( https://www.nfdi4health.de/task-force-covid-19-2 ) within the framework of a DFG-project (LO-342/17-1). LP, VE funded by Mathematical and Statistical modelling project (MUNI/A/1615/2020), Online platform for real-time monitoring, analysis and management of epidemic situations (MUNI/11/02202001/2020); VE also supported by RECETOX research infrastructure (Ministry of Education, Youth and Sports of the Czech Republic: LM2018121), the CETOCOEN EXCELLENCE (CZ.02.1.01/0.0/0.0/17-043/0009632), RECETOX RI project (CZ.02.1.01/0.0/0.0/16-013/0001761). NIB funded by Health Protection Research Unit (grant code NIHR200908). SAb, SF funded by Wellcome Trust (210758/Z/18/Z).