AdCom: Adaptive Combiner for Streaming Aggregations - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2021

AdCom: Adaptive Combiner for Streaming Aggregations

Résumé

Continuous applications such as device monitoring and anomaly detection often require real-time aggregated statistics over unbounded data streams. While existing stream processing systems such as Flink, Spark, and Storm support processing of streaming aggregations, their optimizations are limited with respect to the dynamic nature of the data, and therefore are suboptimal when the workload changes and/or when there is data skew. In this paper we present AdCom, which is an adaptive combiner for stream processing engines. The use of AdCom in aggregation queries enables pre-aggregating tuples upstream (i.e., before data shuffling) followed by global aggregation downstream. In contrast to existing approaches, AdCom can automatically adjust the number of tuples to pre-aggregate depending on the data rate and available network. Our experimental study using real-world streaming workloads shows that using AdCom leads to 2.5-9× higher sustainable throughput without compromising latency.
Fichier principal
Vignette du fichier
main.pdf (905.13 Ko) Télécharger le fichier
Origine : Fichiers éditeurs autorisés sur une archive ouverte

Dates et versions

hal-03156337 , version 1 (02-03-2021)

Identifiants

  • HAL Id : hal-03156337 , version 1

Citer

Felipe Gutierrez, Kaustubh Beedkar, Abel Souza, Volker Markl. AdCom: Adaptive Combiner for Streaming Aggregations. EDBT 2021 - 24th International Conference on Extending Database Technology, Mar 2021, Nicosia, Cyprus. ⟨hal-03156337⟩
121 Consultations
76 Téléchargements

Partager

Gmail Facebook X LinkedIn More