PA-SPS: A predictive adaptive approach for an elastic stream processing system - Archive ouverte HAL
Article Dans Une Revue Journal of Parallel and Distributed Computing Année : 2024

PA-SPS: A predictive adaptive approach for an elastic stream processing system

Résumé

Stream Processing Systems (SPSs) dynamically process input events. Since the input is usually not a constant flow,presenting rate fluctuations, many works in the literature propose to dynamically replicate SPS operators, aiming at reducing the processing bottleneck induced by such fluctuations. However, these SPSs do not consider the problem of load balancing of the replicas or the cost involved in reconfiguring the system whenever the number of replicas changes. We present in this paper a predictive model which, based on input rate variation, execution time of operators, and queued events, dynamically defines the necessary current number of replicas of each operator. A predictor, composed of different models (i.e., mathematical and Machine Learning ones), predicts the input rate. We also propose a Storm-based SPS, named PA-SPS, which uses such a predictive model, not requiring reboot reconfiguration when the number of operators replica change. PA-SPS also implements a load balancer that distributes incoming events evenly among replicas of an operator. We have conducted experiments on Google Cloud Platform (GCP) for evaluation PA-SPS using real traffic traces of different applications and also compared it with Storm and other existing SPSs.
Fichier principal
Vignette du fichier
JPDC-SPS-2024.pdf (775.86 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04664646 , version 1 (30-07-2024)

Identifiants

Citer

Daniel Wladdimiro, Luciana Arantes, Pierre Sens, Nicolás Hidalgo. PA-SPS: A predictive adaptive approach for an elastic stream processing system. Journal of Parallel and Distributed Computing, 2024, 192, pp.104940. ⟨10.1016/j.jpdc.2024.104940⟩. ⟨hal-04664646⟩
68 Consultations
37 Téléchargements

Altmetric

Partager

More