Out-of-Distribution Detection Should Use Conformal Prediction (and Vice-versa?) - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2024

Out-of-Distribution Detection Should Use Conformal Prediction (and Vice-versa?)

Résumé

Research on Out-Of-Distribution (OOD) detection focuses mainly on building scores that efficiently distinguish OOD data from In Distribution (ID) data. On the other hand, Conformal Prediction (CP) uses non-conformity scores to construct prediction sets with probabilistic coverage guarantees. In this work, we propose to use CP to better assess the efficiency of OOD scores. Specifically, we emphasize that in standard OOD benchmark settings, evaluation metrics can be overly optimistic due to the finite sample size of the test dataset. Based on the work of (Bates et al., 2022), we define new conformal AUROC and conformal FRP@TPR95 metrics, which are corrections that provide probabilistic conservativeness guarantees on the variability of these metrics. We show the effect of these corrections on two reference OOD and anomaly detection benchmarks, OpenOOD (Yang et al., 2022) and ADBench (Han et al., 2022). We also show that the benefits of using OOD together with CP apply the other way around by using OOD scores as non-conformity scores, which results in improving upon current CP methods. One of the key messages of these contributions is that since OOD is concerned with designing scores and CP with interpreting these scores, the two fields may be inherently intertwined.
Fichier principal
Vignette du fichier
2403.11532.pdf (1.11 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04559119 , version 1 (25-04-2024)

Identifiants

Citer

Paul Novello, Joseba Dalmau, Léo Andeol. Out-of-Distribution Detection Should Use Conformal Prediction (and Vice-versa?). 2024. ⟨hal-04559119⟩
0 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More