A predictive model for Stream Processing System that dynamically calibrates the number of operator replicas - Archive ouverte HAL Access content directly
Conference Papers Year :

A predictive model for Stream Processing System that dynamically calibrates the number of operator replicas

Abstract

Based on direct acyclic graphs, where vertices and edges respectively correspond to operators and event data flow, Stream Processing Systems (SPSs) are used to process high amounts of data in real time. Operators are allocated in the resources of the infrastructure (e.g., VMs) which are usually replicated for performance sake. We propose in this paper a predictive SPS that dynamically determines the current number of replicas required for each operator based not only on the current resource utilization and data flow variation but also on the events that, due to operator's overloading, could not be processed yet and are, thus, kept in the operator's queue. Preliminary performance results with an application that processes Twitter stream, deployed on Google Cloud Platform (GCP), are presented.
Fichier principal
Vignette du fichier
ComPAS2022_paper_22-2.pdf (242 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-03783768 , version 1 (22-09-2022)

Identifiers

  • HAL Id : hal-03783768 , version 1

Cite

Daniel Wladdimiro, Luciana Arantes, Nicolas Hidalgo, Pierre Sens. A predictive model for Stream Processing System that dynamically calibrates the number of operator replicas. ComPAS 2022 - Conférence francophone d'informatique en Parallélisme, Architecture et Système, Jul 2022, Amiens, France. ⟨hal-03783768⟩
41 View
21 Download

Share

Gmail Facebook Twitter LinkedIn More