A predictive approach for dynamic replication of operators in distributed stream processing systems - Archive ouverte HAL
Communication Dans Un Congrès Année : 2022

A predictive approach for dynamic replication of operators in distributed stream processing systems

Résumé

Stream Processing Systems (SPSs) can present significant fluctuation in input rate. To address this issue, some existing solutions propose reconfiguring the SPS by replicating its operators. However, such reconfiguration usually induces a high system downtime cost. Moreover, reconfiguration decisions are based only on resource utilization without balancing the load between replicas. We propose in this paper a predictive SPS that dynamically defines the necessary number of replicas of each operator based not only on the current resource utilization and input rate variation but also on the events that, due to the operator's overloading, could not be processed yet and are, thus, kept in the operator's queue. In addition, our SPS implements a load balancer that distributes incoming events more evenly among replicas of an operator. Our solution has been integrated into Storm. To avoid system reconfiguration downtime, our SPS preallocates a pool of replicas where each of them can be activated or deactivated based on per operator input load predictions. Using real traffic traces with different applications, we have conducted experiments on Google Cloud Platform (GCP), evaluating our SPS and comparing it with Storm and DABS-Storm.
Fichier principal
Vignette du fichier
SBAC_PAD_2022___Paper___A_predictive_approach_for_dynamic_replication_of_operators_in_distributed_stream_processing_systems.pdf (565.59 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03962939 , version 1 (30-01-2023)

Identifiants

Citer

Daniel Wladdimiro, Luciana Arantes, Pierre Sens, Nicolas Hidalgo. A predictive approach for dynamic replication of operators in distributed stream processing systems. SBAC-PAD 2022 - IEEE 34th International Symposium on Computer Architecture and High Performance Computing, Nov 2022, Bordeaux, France. pp.120-129, ⟨10.1109/SBAC-PAD55451.2022.00023⟩. ⟨hal-03962939⟩
137 Consultations
136 Téléchargements

Altmetric

Partager

More