A scalable and effective rough set theory-based approach for big data pre-processing - Archive ouverte HAL
Article Dans Une Revue Knowledge and Information Systems (KAIS) Année : 2020

A scalable and effective rough set theory-based approach for big data pre-processing

Résumé

A big challenge in the knowledge discovery process is to perform data pre-processing, specifically feature selection, on a large amount of data and high dimensional attribute set. A variety of techniques have been proposed in the literature to deal with this challenge with different degrees of success as most of these techniques need further information about the given input data for thresholding, need to specify noise levels or use some feature ranking procedures. To overcome these limitations, rough set theory (RST) can be used to discover the dependency within the data and reduce the number of attributes enclosed in an input data set while using the data alone and requiring no supplementary information. However, when it comes to massive data sets, RST reaches its limits as it is highly computationally expensive. In this paper, we propose a scalable and effective rough set theory-based approach for large-scale data pre-processing, specifically for feature selection, under the Spark framework. In our detailed experiments, data sets with up to 10,000 attributes have been considered, revealing that our proposed solution achieves a good speedup and performs its feature selection task well without sacrificing performance. Thus, making it relevant to big data.
Fichier principal
Vignette du fichier
ChellyDagdia2020_Article_AScalableAndEffectiveRoughSetT.pdf (2.03 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-04456307 , version 1 (25-06-2020)
hal-04456307 , version 2 (30-05-2024)

Identifiants

Citer

Zaineb Chelly Dagdia, Christine Zarges, Gaël Beck, Mustapha Lebbah. A scalable and effective rough set theory-based approach for big data pre-processing. Knowledge and Information Systems (KAIS), 2020, ⟨10.1007/s10115-020-01467-y⟩. ⟨hal-04456307v1⟩

Collections

GRID5000 SILECS
140 Consultations
171 Téléchargements

Altmetric

Partager

More