Dask-Extended External Tasks for HPC/ML In Transit Workflows
Abstract
In situ workflows are inescapable to fully leverage exascale architectures. They can be complex to build, however, because simulation and data analytics come from two different software ecosystems with their own paradigms and programming models. This work extends the deisa bridging model between MPI+X simulations and distributed task-based analytics; it introduces the concept of external tasks to support the description of analytics graphs spanning multiple timesteps ahead of time while improving scalability. This new approach leads to a straightforward support for contracts between the simulation and analytics graph to limit the data transferred to that actually analyzed in a given execution. We implement this approach using Dask and MPI and evaluate it using an end-toend in-transit workflow that uses an unsupervised ML model for dimensionality reduction. We compare our work with plain Dask postprocessing and with the previous version of deisa. Our work performs better, up to ×7 for the simulation and ×3 for the analytics compared with deisa, and is ×18 less costly compared with plain Dask-all of these with similar development efforts.
Origin | Files produced by the author(s) |
---|