Generate and Update Large HDT RDF Knowledge Graphs on Commodity Hardware - Archive ouverte HAL
Chapitre D'ouvrage Année : 2024

Generate and Update Large HDT RDF Knowledge Graphs on Commodity Hardware

Dennis Diefenbach
Angela Bonifati

Résumé

HDT is a popular compressed file format to store, share and query large RDF Knowledge Graphs (KGs). While all these operations are possible in low hardware settings (i.e. a standard laptop), the generation and updates of HDT files come with an important hardware cost especially in terms of memory and disk usage. In this paper, we present a new tool leveraging HDT, namely k-HDTDiffCat, that allows to a) reduce the memory and disk footprint for the creation of HDT files and to b) remove triples from an existing HDT file thus allowing updates. We show that in a system with 8 times less memory, we can achieve HDT file generation in almost the same time as existing methods. Moreover, our system allows to remove triples from an HDT file catering for updates. This operation is possible without the need to uncompress the original data (as it was the case in the original HDT file) and by keeping low memory consumption. While HDT was suited for storing, exchanging and querying large Knowledge Graphs in low hardware settings, we also offer the novel functionality to generate and update HDT files in these settings. As a side effect, HDT becomes an ideal indexing structure for large KGs in low hardware settings making them more accessible to the community. In particular, we show that we can compress the whole Wikidata graph, which is the largest knowledge graph currently available, on a standard laptop with 16 GB of RAM as well as generate Wikidata indexes that are at most 24 hours behind the live Wikidata endpoint.

Fichier principal
Vignette du fichier
eswc_GenDisk_paper.pdf (738.82 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04769139 , version 1 (06-11-2024)

Identifiants

Citer

Antoine Willerval, Dennis Diefenbach, Angela Bonifati. Generate and Update Large HDT RDF Knowledge Graphs on Commodity Hardware. Albert Meroño Peñuela, Anastasia Dimou, Raphaël Troncy, Olaf Hartig, Maribel Acosta, Mehwish Alam, Heiko Paulheim, Pasquale Lisena. The Semantic Web: 21st International Conference, ESWC 2024, Hersonissos, Crete, Greece, May 26–30, 2024, Proceedings, Part II, 14665, Springer, pp.128-144, 2024, Lecture Notes in Computer Science, 978-3-031-60635-9. ⟨10.1007/978-3-031-60635-9_8⟩. ⟨hal-04769139⟩
12 Consultations
6 Téléchargements

Altmetric

Partager

More