LSH SimilarityJoin Pattern in FastFlow - Archive ouverte HAL Access content directly
Journal Articles International Journal of Parallel Programming Year : 2024

LSH SimilarityJoin Pattern in FastFlow

Abstract

Abstract Similarity joins are recognized to be among the most used data processing and analysis operations. We introduce a C++-based high-level parallel pattern implemented on top of FastFlow Building Blocks to provide the programmer with ready-to-use similarity join computations. The SimilarityJoin pattern is implemented according to the MapReduce paradigm enriched with locality sensitive hashing (LSH) to optimize the whole computation. The new parallel pattern can be used with any C++ serializable data structure and executed on shared- and distributed-memory machines. We present experimental validations of the proposed solution considering two different clusters and small and large input datasets to evaluate in-core and out-of-core executions. The performance assessment of the SimilarityJoin pattern has been conducted by comparing the execution time against the one obtained from the original hand-tuned Hadoop-based implementation of the LSH-based similarity join algorithms as well as a Spark-based version. The experiments show that the SimilarityJoin pattern: (1) offers a significant performance improvement for small and medium datasets; (2) is competitive also for computations using large input datasets producing out-of-core executions.

Dates and versions

hal-04586759 , version 1 (24-05-2024)

Identifiers

Cite

Nicolò Tonci, Sébastien Rivault, Mostafa Bamha, Sophie Robert, Sébastien Limet, et al.. LSH SimilarityJoin Pattern in FastFlow. International Journal of Parallel Programming, 2024, ⟨10.1007/s10766-024-00772-1⟩. ⟨hal-04586759⟩
0 View
0 Download

Altmetric

Share

Gmail Mastodon Facebook X LinkedIn More