n-MeRCI: A new Metric to Evaluate the Correlation Between Predictive Uncertainty and True Error - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2019

n-MeRCI: A new Metric to Evaluate the Correlation Between Predictive Uncertainty and True Error

Michel Moukari
  • Fonction : Auteur
  • PersonId : 1032553
Loïc Simon
Sylvaine Picard
  • Fonction : Auteur
  • PersonId : 1032554

Résumé

As deep learning applications are becoming more and more pervasive in robotics, the question of evaluating the reliability of inferences becomes a central question in the robotics community. This domain, known as predictive uncertainty, has come under the scrutiny of research groups developing Bayesian approaches adapted to deep learning such as Monte Carlo Dropout. Unfortunately, for the time being, the real goal of predictive uncertainty has been swept under the rug. Indeed, these approaches are solely evaluated in terms of raw performance of the network prediction, while the quality of their estimated uncertainty is not assessed. Evaluating such uncertainty prediction quality is especially important in robotics, as actions shall depend on the confidence in perceived information. In this context, the main contribution of this article is to propose a novel metric that is adapted to the evaluation of relative uncertainty assessment and directly applicable to regression with deep neural networks. To experimentally validate this metric, we evaluate it on a toy dataset and then apply it to the task of monocular depth estimation.
Fichier principal
Vignette du fichier
IROS19_1624_MS.pdf (1 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02242792 , version 1 (19-08-2019)

Identifiants

Citer

Michel Moukari, Loïc Simon, Sylvaine Picard, Frédéric Jurie. n-MeRCI: A new Metric to Evaluate the Correlation Between Predictive Uncertainty and True Error. IEEE/RJS International Conference on Intelligent Robots and Systems (IROS), Nov 2019, Macao, China. ⟨hal-02242792⟩
62 Consultations
210 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More