Communication Dans Un Congrès Année : 2025

Assessing the capability of supervised classifiers to recover the true underlying probabilities

Évaluer la capacité des classificateurs supervisés à récupérer les vraies probabilités

Résumé

In classification, a key objective is to produce estimated probabilities that approximate the true underlying probabilities. However, evaluating the quality of these estimates remains challenging since the true probabilities are unobservable.

In this study, we simulate datasets with controlled true distribution shapes to analyze how well different models preserve these distributions. We also assess for these models how well standard calibration metrics (Expected Calibration Error, Brier Score, Log-Loss) reflect the quality of the predicted probabilities.

Our results show that while models like logistic regression, neural networks, and svm provide good probability estimates, others like random forests and naive bayes tend to distort the true probabilities. Importantly, standard calibration metrics such as ECE fail to detect these distortions. In contrast, loss-based metrics such as Log-Loss and Brier Score, especially when decomposed into epistemic and refinement components, offer more informative assessments. These findings highlight that a classifier may appear well calibrated according to ECE, even when its predicted probabilities are misaligned with the true underlying probabilities.

This study emphasizes that when accurate probability estimation is a primary modeling goal, practitioners should rely on loss-based evaluations for comparing models to choose the best one in calibration rather than solely on ECE, and preference should be given to models that better preserve the shape of the true probability distribution.

Fichier principal
Vignette du fichier
Paper ID S1165 Dieye Ndeye Awa AMLDS 2025.pdf (291.35 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-05099239 , version 1 (06-06-2025)

Licence

Identifiants

  • HAL Id : hal-05099239 , version 1

Citer

Ndeye Awa Dieye, Giorgio Russolillo, Ndèye Niang. Assessing the capability of supervised classifiers to recover the true underlying probabilities. International conference on Advanced Machine Learning and Data Science, Jul 2025, Tokyo, Japan. ⟨hal-05099239⟩
261 Consultations
240 Téléchargements

Partager

  • More