Efficiently identifying disguised missing values in heterogeneous, text-rich data - Archive ouverte HAL
Journal Articles Transactions on Large-Scale Data- and Knowledge-Centered Systems Year : 2022

Efficiently identifying disguised missing values in heterogeneous, text-rich data

Abstract

Digital data is produced in many data models, ranging from highly structured (typically relational) to semi-structured models (XML, JSON) to various graph formats (RDF, property graphs) or text. Most real-world datasets contain a certain amount of null values, denoting missing, unknown, or inapplicable information. While some data models allow representing nulls by special tokens, so-called disguised missing values (DMVs, in short) are also frequently encountered: these are values that are not syntactically speaking nulls, but which do, nevertheless, denote the absence, unavailability, or inapplicability of the information. In this work, we tackle the detection of a particular kind of DMV: texts freely entered by human users. This problem is not tackled by DMV detection methods focused on numeric or categoric data; further, it also escapes DMV detection methods based on value frequency, since such free texts are often different from each other, thus most DMVs are unique. We encountered this problem within the ConnectionLens project where heterogeneous data is integrated into large graphs. We present two DMV detection methods for our specific problem: (i) leveraging Information Extraction, already applied in ConnectionLens graphs; and (ii) through text embeddings and classification. We detail their performance-precision trade-offs on real-world datasets.
Fichier principal
Vignette du fichier
Efficiently_identifying_disguised_nulls_in_heterogeneous_text_data__LNCS__final.pdf (1.56 Mo) Télécharger le fichier
Venn-Diagram.jpg (1.2 Mo) Télécharger le fichier
architecture.png (524.14 Ko) Télécharger le fichier
copiright_tldks.pdf (177.02 Ko) Télécharger le fichier
data.png (682.81 Ko) Télécharger le fichier
diagramme_dmv.png (481.71 Ko) Télécharger le fichier
diagramme_dmv2.png (570.98 Ko) Télécharger le fichier
nulls.pdf (158.76 Ko) Télécharger le fichier
reviews.pdf (92.06 Ko) Télécharger le fichier
smartExtract.png (77.26 Ko) Télécharger le fichier
Origin Files produced by the author(s)

Dates and versions

hal-03716868 , version 1 (29-08-2023)

Licence

Copyright

Identifiers

  • HAL Id : hal-03716868 , version 1

Cite

Théo Bouganim, Helena Galhardas, Ioana Manolescu. Efficiently identifying disguised missing values in heterogeneous, text-rich data. Transactions on Large-Scale Data- and Knowledge-Centered Systems, In press, Special Issue on Data Management - Principles, Technologies, and Applications. ⟨hal-03716868⟩
58 View
68 Download

Share

More