iBig Hybrid Architecture for Energy IoT: When the power of Indexing meets Big Data Processing! - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2017

iBig Hybrid Architecture for Energy IoT: When the power of Indexing meets Big Data Processing!

Résumé

Nowadays, IoT data come from multiple sources and a large number of devices. To manage them, IoT frameworks rely on Big Data ecosystems hosted in the cloud to provide scalable storage systems and to achieve scalable processing. Although these ecosystems scale well to process large sizes of data, in many cases this is done naively. Many datasets, such as IoT Energy measurement data, consist, even partially, of attributes that can be indexed to avoid unnecessary and costly data scan at these scales. In this work, we propose the iBig architecture that provides secondary indexing to Big Data processing for energy IoT datasets. Indexes are considered as metadata stored in a separate component integrated in the ecosystem. Subsequently, MapReduce-based and MPP (massively parallel processing)-based processing leverage indexing to handle only relevant data in a given dataset. Our experimental evaluation on the Grid5000 cloud testbed demonstrates that performance gains can exceed 98% for the MapReduce-based Spark and 81% for the MPP-based Drill for Energy IoT Data. Furthermore, we provide comparison insights about the Spark and Drill frameworks when processing the whole dataset or only with relevant data.
Fichier principal
Vignette du fichier
cloudcom.pdf (868.97 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01705607 , version 1 (09-02-2018)

Identifiants

  • HAL Id : hal-01705607 , version 1

Citer

Houssem-Eddine Chihoub, Christine Collet. iBig Hybrid Architecture for Energy IoT: When the power of Indexing meets Big Data Processing!. The IEEE International Conference on Cloud Computing Technology & Science 2017, Dec 2017, Hong Kong, Hong Kong SAR China. ⟨hal-01705607⟩
62 Consultations
163 Téléchargements

Partager

Gmail Facebook X LinkedIn More