GUESS: monitorinG join qUery Execution in Serverless and Serverful spark
Résumé
This paper proposes a monitoring system called GUESS to
compare the performance and energy consumption of join query processing
on Spark in Serverless and Serverful environments. The system
collects metrics on resource utilization, query execution times, and power
usage through Prometheus, Grafana, Spark History Server, and Open-
Manage Enterprise Power Manager. These metrics are visualized through
an intuitive web dashboard to enable easy comparison between Serverless
and Serverful Spark workloads. Experimental results using the TPC-H
benchmark show that the Serverless environment consumes less energy
than the Serverful environment due to on-demand resource allocation.
However, the Serverful environment exhibits better query performance,
especially for workloads with known resource requirements. GUESS provides
insights into optimizing resource efficiency and query performance
when deploying Spark analytic workloads.
Origine | Fichiers produits par l'(les) auteur(s) |
---|