StreamBed: Capacity Planning for Stream Processing - Archive ouverte HAL
Communication Dans Un Congrès Année : 2024

StreamBed: Capacity Planning for Stream Processing

Résumé

We present StreamBed, a capacity planning system for stream processing. StreamBed predicts, ahead of any production deployment, the resources that a query will require to process an incoming data rate sustainably, and the appropriate configuration of these resources. For this purpose, StreamBed builds a capacity planning model by piloting a series of runs of the target query in a small-scale, controlled testbed. We implement StreamBed for Apache Flink. Our evaluation with large-scale queries of the Nexmark benchmark demonstrates that StreamBed can accurately predict capacity requirements for jobs spanning more than 1,000 cores using a model built with a 48-core testbed.
Fichier principal
Vignette du fichier
FINAL.pdf (2.82 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04708354 , version 1 (24-09-2024)

Licence

Identifiants

Citer

Guillaume Rosinosky, Donatien Schmitz, Etienne Rivière. StreamBed: Capacity Planning for Stream Processing. DEBS 2024 - 18th ACM International Conference on Distributed and Event-based Systems, Jun 2024, Lyon, France. pp.90-102, ⟨10.1145/3629104.3666034⟩. ⟨hal-04708354⟩
24 Consultations
17 Téléchargements

Altmetric

Partager

More