Abstract Each International Geomagnetic Reference Field (IGRF) model released under the auspices of the International Association of Geomagnetism and Aeronomy comprises a secular variation component that describes the evolution of the main magnetic field anticipated for the 5 years to come. Every Gauss coefficient, up to spherical harmonic degree and order 8, is assumed to undergo its own independent linear evolution. With a mathematical model of the core magnetic field and its time rate of change constructed from geomagnetic observations at hand, a standard prediction of the secular variation (SV) consists of taking the time rate of change of each Gauss coefficient at the final time of analysis as the predicted rate of change. The last three generations of the IGRF have additionally witnessed a growing number of candidate SV models relying upon physics-based forecasts. This surge is motivated by satellite data that now span more than two decades and by the concurrent progress in the numerical modelling of Earth’s core dynamics. Satellite data reveal rapid (interannual) geomagnetic features whose imprint can be detrimental to the quality of the IGRF prediction. This calls for forecasting frameworks able to incorporate at least part of the processes responsible for short-term geomagnetic variations. In this letter, we perform a retrospective analysis of the performance of past IGRF SV models and candidates over the past 35 years; we emphasize that over the satellite era, the quality of the 5-year forecasts worsens at times of rapid geomagnetic changes. After the definition of the time scales that are relevant for the IGRF prediction exercise, we cover the strategies followed by past physics-based candidates, which we categorize into a “‘core–surface flow” family and a “dynamo” family, noting that both strategies resort to “input” models of the main field and its secular variation constructed from observations. We next review practical lessons learned from our previous attempts. Finally, we discuss possible improvements on the current state of affairs in two directions: the feasibility of incorporating rapid physical processes into the analysis on the one hand, and the accuracy and quantification of the uncertainty impacting input models on the other hand.