Toward Green Data Lake Management and Analysis through a CTMC Model
Résumé
In the realm of big data era, large-scale data processing systems are considered critical energy-consumption sources that exert significant impacts on the ecosystem. With the emergence of big data environments, which leads to the production of multi-structured data, numerous solutions and technologies have arisen to efficiently process vast amounts of data, to capitalize on real-time data generation. Recently, a centralized data repository platform called Data Lake has been proposed to manage the heterogeneous data originating from diverse sources, and sustainability issues in data platform management have gained traction. In this article, we intend to design and analyze an energy-aware strategy for a green data lake framework. This strategy prioritizes ecological concerns, particularly energy consumption, ensuring the maintenance of sufficient quality of service in terms of data availability. We present a model based on continuous-time Markov chain (CTMC) to analyze the trade-off between energy efficiency and performance of a green data lake platform.