Data Management for the RedisDG Scientific Workflow Engine - Archive ouverte HAL
Communication Dans Un Congrès Année : 2016

Data Management for the RedisDG Scientific Workflow Engine

Résumé

—In this paper we investigate the general problem of controlling a scientific workflow service in terms of data management. We focus on the data management problem for the RedisDG scientific workflow engine. RedisDG is based on the Publish/Subscribe paradigm for the interaction between the different components of the system, hence new issues appear for scheduling. Indeed, the Publish/Subscribe paradigm utilization introduces different challenging problems, among them the design of effective solutions for managing data, on the fly, when tasks are published. Our contributions are twofold. First we add new functionalities to the RedisDG workflow engine with scheduling decisions related to the allocation of data intensive jobs to compute units and according to an efficient management of data and second we introduce a large set of experiments to validate our approaches. We analyze our results and we also sketch perspectives and insights. Experiments are conducted on the Grid'5000 testbed and the paper is a step forward to implement a 'Workflow engine as a Service' (WaaS).
Fichier principal
Vignette du fichier
RedisDG_SC2_2016.pdf (560.74 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01517163 , version 1 (02-05-2017)

Identifiants

Citer

Leila Abidi, Souha Bejaoui, Christophe Cérin, Jonathan Lejeune, Yanik Ngoko, et al.. Data Management for the RedisDG Scientific Workflow Engine. IEEE International Conference on Computer and Information Technology, Dec 2016, Nadi, Fiji. pp.599 - 606, ⟨10.1109/CIT.2016.55⟩. ⟨hal-01517163⟩
344 Consultations
222 Téléchargements

Altmetric

Partager

More