Deep Learning Uncertainty in Machine Teaching - Archive ouverte HAL Access content directly
Conference Papers Year : 2022

Deep Learning Uncertainty in Machine Teaching

Abstract

Machine Learning models can output confident but incorrect predictions. To address this, ML researchers use various techniques to reliably estimate ML uncertainty, usually performed on controlled benchmarks once the model has been trained. We explore how the two types of uncertainty-aleatoric and epistemic-can help non-expert users understand the strengths and weaknesses of a classifier in an interactive setting. We are interested not only in their use of uncertainty to teach and understand the classifier, but also in their perception of the difference between aleatoric and epistemic uncertainty. We conducted an experiment where non-experts train a classifier to recognize card images, and are tested on their ability to predict classifier outcomes. Participants who used either larger or more varied training sets significantly improved their understanding of uncertainty, both epistemic or aleatoric. However, participants who relied on the uncertainty measure to guide their choice of training data did not significantly improve classifier training, nor were they better able to guess the classifier outcome. We identified three specific situations where participants successfully identified the difference between aleatoric and epistemic uncertainty: placing a new card in the exact same position as a training card; placing different cards next to each other; and placing a non-card, such as their hand, next to or on top of a card. We discuss our methodology for estimating uncertainty for Interactive Machine Learning systems and question the need for two-level uncertainty in Machine Teaching.
Fichier principal
Vignette du fichier
iui_2022_author_version.pdf (13.84 Mo) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03579448 , version 1 (18-02-2022)

Identifiers

Cite

Téo Sanchez, Baptiste Caramiaux, Pierre Thiel, Wendy E. Mackay. Deep Learning Uncertainty in Machine Teaching. IUI 2022 - 27th Annual Conference on Intelligent User Interfaces, Mar 2022, Helsinki / Virtual, Finland. ⟨10.1145/3490099.3511117⟩. ⟨hal-03579448⟩
518 View
311 Download

Altmetric

Share

Gmail Facebook X LinkedIn More