Efficient Execution of Scientific Workflows in the Cloud Through Adaptive Caching - Archive ouverte HAL
Article Dans Une Revue Transactions on Large-Scale Data- and Knowledge-Centered Systems Année : 2020

Efficient Execution of Scientific Workflows in the Cloud Through Adaptive Caching

Résumé

Many scientific experiments are now carried on using scientific workflows, which are becoming more and more data-intensive and complex. We consider the efficient execution of such workflows in the cloud. Since it is common for workflow users to reuse other workflows or data generated by other workflows, a promising approach for efficient workflow execution is to cache intermediate data and exploit it to avoid task re-execution. In this paper, we propose an adaptive caching solution for data-intensive workflows in the cloud. Our solution is based on a new scientific workflow management architecture that automatically manages the storage and reuse of intermediate data and adapts to the variations in task execution times and output data size. We evaluated our solution by implementing it in the OpenAlea system and performing extensive experiments on real data with a data-intensive application in plant phenotyping. The results show that adaptive caching can yield major performance gains, e.g., up to a factor of 3.5 with 6 workflow re-executions.
Fichier principal
Vignette du fichier
TLDKS_2020(1).pdf (4.61 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02969510 , version 1 (16-10-2020)

Identifiants

Citer

Gaëtan Heidsieck, Daniel de Oliveira, Esther Pacitti, Christophe Pradal, Francois Tardieu, et al.. Efficient Execution of Scientific Workflows in the Cloud Through Adaptive Caching. Transactions on Large-Scale Data- and Knowledge-Centered Systems, 2020, pp.41-66. ⟨10.1007/978-3-662-62271-1_2⟩. ⟨hal-02969510⟩
158 Consultations
137 Téléchargements

Altmetric

Partager

More