Confidence intervals uncovered: Are we ready for real-world medical imaging AI? - Archive ouverte HAL
Communication Dans Un Congrès Année : 2024

Confidence intervals uncovered: Are we ready for real-world medical imaging AI?

Nicola Rieke
  • Fonction : Auteur

Résumé

Medical imaging is spearheading the AI transformation of healthcare. Performance reporting is key to determine which methods should be translated into clinical practice. Frequently, broad conclusions are simply derived from mean performance values. In this paper, we argue that this common practice is often a misleading simplification as it ignores performance variability. Our contribution is threefold. (1) Analyzing all MICCAI segmentation papers (n = 221) published in 2023, we first observe that more than 50% of papers do not assess performance variability at all. Moreover, only one (0.5%) paper reported confidence intervals (CIs) for model performance. (2) To address the reporting bottleneck, we show that the unreported standard deviation (SD) in segmentation papers can be approximated by a second-order polynomial function of the mean Dice similarity coefficient (DSC). Based on external validation data from 56 previous MICCAI challenges, we demonstrate that this approximation can accurately reconstruct the CI of a method using information provided in publications. (3) Finally, we reconstructed 95% CIs around the mean DSC of MICCAI 2023 segmentation papers. The median CI width was 0.03 which is three times larger than the median performance gap between the first and second ranked method. For more than 60% of papers, the mean performance of the second-ranked method was within the CI of the first-ranked method. We conclude that current publications typically do not provide sufficient evidence to support which models could potentially be translated into clinical practice.
Fichier principal
Vignette du fichier
merged.pdf (4.85 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Licence

Dates et versions

hal-04715638 , version 1 (01-10-2024)

Licence

Identifiants

  • HAL Id : hal-04715638 , version 1

Citer

Evangelia Christodoulou, Annika Reinke, Rola Houhou, Piotr Kalinowski, Selen Erkan, et al.. Confidence intervals uncovered: Are we ready for real-world medical imaging AI?. MICCAI 2024 - Medical Image Computing and Computer-Assisted Intervention, Oct 2024, Marrakech, Morocco. ⟨hal-04715638⟩
0 Consultations
0 Téléchargements

Partager

More