Optimization for Large-Scale Fuzzy Joins Using Fuzzy Filters in MapReduce - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2020

Optimization for Large-Scale Fuzzy Joins Using Fuzzy Filters in MapReduce

Résumé

A fuzzy or similarity join is one of the most useful data processing and analysis operations for Big Data in a general context. It combines pairs of tuples for which the distance is lower than or equal to a given threshold ε. The fuzzy join is used in many practical applications, but it is extremely costly in time and space, and may even not be executed on large-scale datasets. Although there have been some studies to improve its performance by applying filters, a solution of an effective fuzzy filter for the join has never been conducted. In this paper, we thus extend our previous work by proposing a novel fuzzy filter to optimize fuzzy joins. This filter is a compact, probabilistic data structure that supports very fast similarity queries by maintaining a bit matrix, with small false positive rate and zero false negative rate. We show that our proposal is more efficient than others because of eliminating redundant data, reducing computation cost and avoiding duplicate output.
Fichier principal
Vignette du fichier
fuzzieee2020.pdf (1.14 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03107555 , version 1 (12-01-2021)

Identifiants

Citer

Thi-To-Quyen Tran, Thuong-Cang Phan, Anne Laurent, Laurent d'Orazio. Optimization for Large-Scale Fuzzy Joins Using Fuzzy Filters in MapReduce. FUZZ-IEEE 2020 - International Conference on Fuzzy Systems, Jul 2020, Glasgow, United Kingdom. pp.1-8, ⟨10.1109/FUZZ48607.2020.9177610⟩. ⟨hal-03107555⟩
87 Consultations
175 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More