Dask-Extended External Tasks for HPC/ML In Transit Workflows - DATAMOVE - Mouvement de données pour le calcul haute performance
Communication Dans Un Congrès Année : 2023

Dask-Extended External Tasks for HPC/ML In Transit Workflows

Amal Gueroudji
Julien Bigot
Bruno Raffin
Robert Ross

Résumé

In situ workflows are inescapable to fully leverage exascale architectures. They can be complex to build, however, because simulation and data analytics come from two different software ecosystems with their own paradigms and programming models. This work extends the deisa bridging model between MPI+X simulations and distributed task-based analytics; it introduces the concept of external tasks to support the description of analytics graphs spanning multiple timesteps ahead of time while improving scalability. This new approach leads to a straightforward support for contracts between the simulation and analytics graph to limit the data transferred to that actually analyzed in a given execution. We implement this approach using Dask and MPI and evaluate it using an end-toend in-transit workflow that uses an unsupervised ML model for dimensionality reduction. We compare our work with plain Dask postprocessing and with the previous version of deisa. Our work performs better, up to ×7 for the simulation and ×3 for the analytics compared with deisa, and is ×18 less costly compared with plain Dask-all of these with similar development efforts.
Fichier principal
Vignette du fichier
deisa.pdf (1.09 Mo) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04409157 , version 1 (22-01-2024)

Licence

Identifiants

Citer

Amal Gueroudji, Julien Bigot, Bruno Raffin, Robert Ross. Dask-Extended External Tasks for HPC/ML In Transit Workflows. SC-W 2023 - Workshops of The International Conference on High Performance Computing, Network, Storage, and Analysis, Nov 2023, Denver, United States. pp.831-838, ⟨10.1145/3624062.3624151⟩. ⟨hal-04409157⟩
173 Consultations
85 Téléchargements

Altmetric

Partager

More