Analytics on Non-Normalized Data Sources: more Learning, rather than more Cleaning - Archive ouverte HAL
Article Dans Une Revue IEEE Access Année : 2022

Analytics on Non-Normalized Data Sources: more Learning, rather than more Cleaning

Résumé

Data analysis is increasingly performed over data assembled from uncontrolled sources, facing inconsistency in knowledge-representation conventions. The typical practice is to create "clean" data for analysis, matching entities and merging variants to overcome differences in knowledge representation. Despite progress in data management techniques to automate this process, it still needs labor-intensive supervision from the analyst. In this paper, we evaluate the benefit of advanced statistical tools to address directly many analytic tasks across data sources without such entity-matching cleaning. Reframing analytical questions as machine-learning tasks enables to replace exact matching of entities by continuous descriptions-vectorial embeddings-that expose similarities between entries. But are analyses with less cleaning trustworthy? We answer this question with a thorough benchmark on questions typical of socioeconomic studies across 14 employee databases: we compare the approaches based on machine learning to manual data cleaning (entity matching). It reveals that using embeddings and machine learning improves results validity (smaller estimation error) more than manual cleaning, with considerably less human labor. While machine learning is often combined with data management for the purpose of cleaning, our study suggests that using it directly for analysis is beneficial because it captures ambiguities hard to represent during curation.
Fichier principal
Vignette du fichier
final.pdf (3.98 Mo) Télécharger le fichier
Origine Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-03647434 , version 1 (20-04-2022)
hal-03647434 , version 2 (25-04-2022)

Identifiants

Citer

Alexis Cvetkov-Iliev, Alexandre Allauzen, Gaël Varoquaux. Analytics on Non-Normalized Data Sources: more Learning, rather than more Cleaning. IEEE Access, inPress, 10, pp.42420-42431. ⟨10.1109/ACCESS.2022.3168013⟩. ⟨hal-03647434v2⟩
209 Consultations
207 Téléchargements

Altmetric

Partager

More