Explainability has been at the forefront of machine learning research in recent years. Despite large volumes of research already conducted, a consensus on what should be explained and in what contexts is still lacking. Perhaps a generic consensus is not even possible. Our position is to bring forward explanations of the machine learning process rather than, or in addition to, explaining machine-learned outcomes. In most practical tasks for machine learning, evaluation criteria often evolve during the modelling process. Many model variants are tested before the final model is selected if it is ever final. Explainability research should pay closer attention to explaining the optimisation criteria used for model fitting, evaluation, and selection in more realistic ways.