LowFive: In Situ Data Transport for High-Performance Workflows - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2023

LowFive: In Situ Data Transport for High-Performance Workflows

Résumé

We describe LowFive, a new data transport layer based on the HDF5 data model, for in situ workflows. Executables using LowFive can communicate in situ (using in-memory data and MPI message passing), reading and writing traditional HDF5 files to physical storage, and combining the two modes. Minimal and often no source-code modification is needed for programs that already use HDF5. LowFive maintains deep copies or shallow references of datasets, configurable by the user. More than one task can produce (write) data, and more than one task can consume (read) data, accommodating fan-in and fan-out in the workflow task graph. LowFive supports data redistribution from n producer processes to m consumer processes. We demonstrate the above features in a series of experiments featuring both synthetic benchmarks as well as a representative use case from a scientific workflow, and we also compare with other data transport solutions in the literature.
Fichier principal
Vignette du fichier
lowfive-ipdps23.pdf (1.3 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04119925 , version 1 (07-06-2023)

Licence

Paternité

Identifiants

  • HAL Id : hal-04119925 , version 1

Citer

Tom Peterka, Dmitriy Morozov, Arnur Nigmetov, Orcun Yildiz, Bogdan Nicolae, et al.. LowFive: In Situ Data Transport for High-Performance Workflows. IPDPS'23: The 37th IEEE International Parallel and Distributed Processing Symposium, ACM; IEEE, May 2023, St.Petersburg, United States. ⟨hal-04119925⟩
30 Consultations
109 Téléchargements

Partager

Gmail Facebook X LinkedIn More