Deduplication Over Heterogeneous Attribute Types (D-HAT) - Archive ouverte HAL
Communication Dans Un Congrès Année : 2022

Deduplication Over Heterogeneous Attribute Types (D-HAT)

Résumé

Deduplication is the task of recognizing multiple representations of the same real-world object. The majority of existing solutions focuses on textual data, this means that data sets containing boolean and numerical attribute types are rarely considered in the literature, while the problem of missing values is inadequately covered. Supervised solutions cannot be applied without an adequate number of labelled examples, but training data for deduplication can only be obtained through time-costly processes. In high dimensional data sets, feature engineering is also required to avoid the risk of overfitting. To address these challenges, we go beyond existing works through D-HAT, a clustering-based pipeline that is inherently capable of handling high dimensional, sparse and heterogeneous attribute types. At its core lies: (i) a novel matching function that effectively summarizes multiple matching signals, and (ii) MutMax, a greedy clustering algorithm that designates as duplicates the pairs with a mutually maximum matching score. We evaluate D-HAT on five established, real-world benchmark data sets, demonstrating that our approach outperforms the state-of-the-art supervised and unsupervised deduplication algorithms to a significant extent.
Fichier principal
Vignette du fichier
DeduplicationOverHeterogeneousAttributeTypesD-HAT-Liekah-Papadakis.pdf (618.87 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04097219 , version 1 (15-05-2023)

Identifiants

Citer

Loujain Liekah, George Papadakis. Deduplication Over Heterogeneous Attribute Types (D-HAT). Advanced Data Mining and Applications, Nov 2022, Brisbane, Australia. pp.379-391, ⟨10.1007/978-3-031-22137-8_28⟩. ⟨hal-04097219⟩
30 Consultations
83 Téléchargements

Altmetric

Partager

More