Efficiently identifying disguised missing values in heterogeneous, text-rich data
Résumé
Digital data is produced in many data models, ranging from highly structured (typically relational) to semi-structured models (XML, JSON) to various graph formats (RDF, property graphs) or text. Most real-world datasets contain a certain amount of null values, denoting missing, unknown, or inapplicable information. While some data models allow representing nulls by special tokens, so-called disguised missing values (DMVs, in short) are also frequently encountered: these are values that are not syntactically speaking nulls, but which do, nevertheless, denote the absence, unavailability, or inapplicability of the information.
In this work, we tackle the detection of a particular kind of DMV: texts freely entered by human users. This problem is not tackled by DMV detection methods focused on numeric or categoric data; further, it also escapes DMV detection methods based on value frequency, since such free texts are often different from each other, thus most DMVs are unique. We encountered this problem within the ConnectionLens project where heterogeneous data is integrated into large graphs. We present two DMV detection methods for our specific problem: (i) leveraging Information Extraction, already applied in ConnectionLens graphs; and (ii) through text embeddings and classification. We detail their performance-precision trade-offs on real-world datasets.
Domaines
Base de données [cs.DB]
Fichier principal
Efficiently_identifying_disguised_nulls_in_heterogeneous_text_data__LNCS__final.pdf (1.56 Mo)
Télécharger le fichier
Venn-Diagram.jpg (1.2 Mo)
Télécharger le fichier
architecture.png (524.14 Ko)
Télécharger le fichier
copiright_tldks.pdf (177.02 Ko)
Télécharger le fichier
data.png (682.81 Ko)
Télécharger le fichier
diagramme_dmv.png (481.71 Ko)
Télécharger le fichier
diagramme_dmv2.png (570.98 Ko)
Télécharger le fichier
nulls.pdf (158.76 Ko)
Télécharger le fichier
reviews.pdf (92.06 Ko)
Télécharger le fichier
smartExtract.png (77.26 Ko)
Télécharger le fichier
| Origine | Fichiers produits par l'(les) auteur(s) |
|---|