A predictive model for Stream Processing System that dynamically calibrates the number of operator replicas - Archive ouverte HAL
Communication Dans Un Congrès Année : 2022

A predictive model for Stream Processing System that dynamically calibrates the number of operator replicas

Résumé

Based on direct acyclic graphs, where vertices and edges respectively correspond to operators and event data flow, Stream Processing Systems (SPSs) are used to process high amounts of data in real time. Operators are allocated in the resources of the infrastructure (e.g., VMs) which are usually replicated for performance sake. We propose in this paper a predictive SPS that dynamically determines the current number of replicas required for each operator based not only on the current resource utilization and data flow variation but also on the events that, due to operator's overloading, could not be processed yet and are, thus, kept in the operator's queue. Preliminary performance results with an application that processes Twitter stream, deployed on Google Cloud Platform (GCP), are presented.
Fichier principal
Vignette du fichier
ComPAS2022_paper_22-2.pdf (242 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03783768 , version 1 (22-09-2022)

Identifiants

  • HAL Id : hal-03783768 , version 1

Citer

Daniel Wladdimiro, Luciana Arantes, Nicolas Hidalgo, Pierre Sens. A predictive model for Stream Processing System that dynamically calibrates the number of operator replicas. ComPAS 2022 - Conférence francophone d'informatique en Parallélisme, Architecture et Système, Jul 2022, Amiens, France. ⟨hal-03783768⟩
131 Consultations
83 Téléchargements

Partager

More