Exploiting Formal Concept Analysis for Data Modeling in Data Lakes - Archive ouverte HAL
Communication Dans Un Congrès Année : 2024

Exploiting Formal Concept Analysis for Data Modeling in Data Lakes

Anes Bendimerad
Romain Mathonat
Youcef Remil
Mehdi Kaytoue

Résumé

Data lakes are widely used to store extensive and heterogeneous datasets for advanced analytics. However, the unstructured nature of data in these repositories introduces complexities in exploiting them and extracting meaningful insights. This motivates the need of exploring efficient approaches for consolidating data lakes and deriving a common and unified schema. This paper introduces a practical data visualization and analysis approach rooted in Formal Concept Analysis (FCA) to systematically clean, organize, and design data structures within a data lake. We explore diverse data structures stored in our data lake at Infologic, including InfluxDB measurements and Elasticsearch indexes, aiming to derive conventions for a more accessible data model. Leveraging FCA, we represent data structures as objects, analyze the concept lattice, and present two strategies-top-down and bottom-up-to unify these structures and establish a common schema. Our methodology yields significant results, enabling the identification of common concepts in the data structures, such as resources along with their underlying shared fields (timestamp, type, usedRatio, etc.). Moreover, the number of distinct data structure field names is reduced by 54 percent (from 190 to 88) in the studied subset of our data lake. We achieve a complete coverage of 80 percent of data structures with only 34 distinct field names, a significant improvement from the initial 121 field names that were needed to reach such coverage. The paper provides insights into the Infologic ecosystem, problem formulation, exploration strategies, and presents both qualitative and quantitative results.The source code and datasets of this work are made available: https://zenodo.org/records/10589722
Fichier principal
Vignette du fichier
_Camera_ready__Exploiting_FCA_for_data_modeling_in_data_lakes.pdf (1.62 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04670148 , version 1 (11-08-2024)

Identifiants

Citer

Anes Bendimerad, Romain Mathonat, Youcef Remil, Mehdi Kaytoue. Exploiting Formal Concept Analysis for Data Modeling in Data Lakes. 1st International Joint Conference on Conceptual Knowledge Structures (CONCEPTS 2024), Sep 2024, Cadix, Spain. pp.270-285, ⟨10.1007/978-3-031-67868-4_18⟩. ⟨hal-04670148⟩
66 Consultations
33 Téléchargements

Altmetric

Partager

More