Calibration in Machine Learning Uncertainty Quantification: beyond consistency to target adaptivity
Résumé
Reliable uncertainty quantification (UQ) in machine learning (ML) regression tasks
is becoming the focus of many studies in materials and chemical science. It is now
well understood that average calibration is insufficient, and most studies implement
additional methods testing the conditional calibration with respect to uncertainty, i.e.
consistency. Consistency is assessed mostly by so-called reliability diagrams. There
exists however another way beyond average calibration, which is conditional calibra-
tion with respect to input features, i.e. adaptivity. In practice, adaptivity is the main
concern of the final users of a ML-UQ method, seeking for the reliability of predic-
tions and uncertainties for any point in features space. This article aims to show that
consistency and adaptivity are complementary validation targets, and that a good
consistency does not imply a good adaptivity. An integrated validation framework
is proposed and illustrated on a representative example.
Origine : Fichiers produits par l'(les) auteur(s)
Licence : CC BY ND - Paternité - Pas de modifications
Licence : CC BY ND - Paternité - Pas de modifications