Interactive anomaly detection in mixed tabular data using Bayesian networks - Archive ouverte HAL Access content directly
Conference Papers Year : 2020

Interactive anomaly detection in mixed tabular data using Bayesian networks

Abstract

The last decades improvements in processing abilities have quickly led to an increasing use of data analyses implying massive data-sets. To retrieve insightful information from any data driven approach, a pivotal aspect to ensure is good data quality. Manual correction of massive data-sets requires tremendous efforts, is prone to errors, and results being really costly. If knowledge in a specific field can often allow the development of efficient models for anomaly detection and data correction, this knowledge can sometimes be unavailable and a more generic approach should be found. This paper presents a novel approach to anomaly detection and correction in mixed tabular data using Bayesian Networks. We present an algorithm for detecting anomalies and offering correction hints based on Jensen scores computed within the Markov Blankets of considered variables. We also discuss the incremental corrections of detection model using user's feedback, as well as additional aspects related to discretization in mixed data and its effects on detection efficiency. Finally we also discuss how functional dependencies can be managed to detect errors while improving faithfulness and computation speed.
Fichier principal
Vignette du fichier
dufraisse20a.pdf (451.08 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03014622 , version 1 (17-03-2021)

Identifiers

  • HAL Id : hal-03014622 , version 1

Cite

Evan Dufraisse, Philippe Leray, Raphaël Nedellec, Tarek Benkhelif. Interactive anomaly detection in mixed tabular data using Bayesian networks. 10th International Conference on Probabilistic Graphical Models (PGM 2020), Sep 2020, Aalborg, Denmark. ⟨hal-03014622⟩
159 View
219 Download

Share

Gmail Mastodon Facebook X LinkedIn More