Communication Dans Un Congrès Année : 2025

Decision from Suboptimal Classifiers: Excess Risk Pre-and Post-Calibration

Résumé

Probabilistic classifiers are central for making informed decisions under uncertainty. Based on the maximum expected utility principle, optimal decision rules can be derived using the posterior class probabilities and misclassification costs. Yet, in practice only learned approximations of the oracle posterior probabilities are available. In this work, we quantify the excess risk (a.k.a. regret) incurred using approximate posterior probabilities in batch binary decision-making. We provide analytical expressions for miscalibration-induced regret (R CL ), as well as tight and informative upper and lower bounds on the regret of calibrated classifiers (R GL ). These expressions allow us to identify regimes where recalibration alone addresses most of the regret, and regimes where the regret is dominated by the grouping loss, which calls for post-training beyond recalibration. Crucially, both R CL and R GL can be estimated in practice using a calibration curve and a recent grouping loss estimator. On NLP experiments, we show that these quantities identify when the expected gain of more advanced post-training is worth the operational cost. Finally, we highlight the potential of multicalibration approaches as efficient alternatives to costlier fine-tuning approaches.

Fichier principal
Vignette du fichier
main.pdf (3.06 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04987054 , version 1 (12-03-2025)

Licence

Identifiants

  • HAL Id : hal-04987054 , version 1

Citer

Alexandre Perez-Lebel, Gaël Varoquaux, Sanmi Koyejo, Matthieu Doutreligne, Marine Le Morvan. Decision from Suboptimal Classifiers: Excess Risk Pre-and Post-Calibration. AISTATS 2025 - the 28th International Conference on Artifi- cial Intelligence and Statistics, May 2025, Mai Khao, Thailand. ⟨hal-04987054⟩
165 Consultations
133 Téléchargements

Partager

  • More