LSH SimilarityJoin pattern in FastFlow
Résumé
Similarity joins are recognized to be among the most used data
processing and analysis operations. In this work we introduce a C++-based
high-level parallel pattern implemented on top of FastFlow Building Blocks to
provide the programmer with ready-to-use similarity joins computations. The
SimilarityJoin pattern is implemented according to the MapReduce paradigm
enriched with Locality Sensitive Hashing (LSH) to optimize the whole com-
putation. The new parallel pattern can be used with any C++ serializable
data structure and executed on shared- and distributed-memory machines.
We present some experimental validation of the proposed solution on two dif-
ferent clusters using the original hand-tuned Hadoop implementation of the
LSH-based similarity join algorithms as a reference baseline.