LSH SimilarityJoin Pattern in FastFlow - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue International Journal of Parallel Programming Année : 2024

LSH SimilarityJoin Pattern in FastFlow

Résumé

Abstract Similarity joins are recognized to be among the most used data processing and analysis operations. We introduce a C++-based high-level parallel pattern implemented on top of FastFlow Building Blocks to provide the programmer with ready-to-use similarity join computations. The SimilarityJoin pattern is implemented according to the MapReduce paradigm enriched with locality sensitive hashing (LSH) to optimize the whole computation. The new parallel pattern can be used with any C++ serializable data structure and executed on shared- and distributed-memory machines. We present experimental validations of the proposed solution considering two different clusters and small and large input datasets to evaluate in-core and out-of-core executions. The performance assessment of the SimilarityJoin pattern has been conducted by comparing the execution time against the one obtained from the original hand-tuned Hadoop-based implementation of the LSH-based similarity join algorithms as well as a Spark-based version. The experiments show that the SimilarityJoin pattern: (1) offers a significant performance improvement for small and medium datasets; (2) is competitive also for computations using large input datasets producing out-of-core executions.

Dates et versions

hal-04586759 , version 1 (24-05-2024)

Identifiants

Citer

Nicolò Tonci, Sébastien Rivault, Mostafa Bamha, Sophie Robert, Sébastien Limet, et al.. LSH SimilarityJoin Pattern in FastFlow. International Journal of Parallel Programming, 2024, ⟨10.1007/s10766-024-00772-1⟩. ⟨hal-04586759⟩
1 Consultations
0 Téléchargements

Altmetric

Partager

Gmail Mastodon Facebook X LinkedIn More