A Flexible and Distributed Runtime System for High-Throughput Constrained Data Streams Generation - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2019

A Flexible and Distributed Runtime System for High-Throughput Constrained Data Streams Generation

Résumé

Major research topics on parallel and distributed frameworks focus on reliability, performance and programmability of large scale systems for, e.g., HPC or Big Data. The solutions proposed are often directly impacted by the large scale nature of the problems. Differently, high-throughput data stream generation is an important challenge for many scientific and industrial applications which is typically well suited for small to medium scale systems, and which has to respect specific constraints about, e.g., speed, throughput or output location. In this paper we present a framework dedicated to this class of problems. We propose a performance-oriented runtime system architecture able to generate constrained data streams issued from jobs dynamically submitted by the user. Our architecture is designed to scale from a single host to a medium-sized cluster with large topology flexibility to achieve high throughput capabilities while being widely adaptive to a variety of problems. We provide experimental evidence of the ability of our framework to meet high-throughput constraints on an industrial use-case, i.e., professional digital printing, that may require tens of Gbit/s sustained output rates. We show in our measurements that our system scales and reaches data rates close to the maximum throughput of our experimental cluster.
Fichier principal
Vignette du fichier
A_Flexible_Runtime_System_for_High_Throughput_Constrained_Data_Stream_Generation.pdf (503.4 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-02381750 , version 1 (26-06-2020)

Identifiants

Citer

Paul Godard, Vincent Loechner, Cédric Bastoul, Frederic Soulier, Guillaume Muller. A Flexible and Distributed Runtime System for High-Throughput Constrained Data Streams Generation. IPDPSW 2019 - IEEE International Symposium on Parallel and Distributed Processing, Workshops and Phd Forum, May 2019, Rio de Janeiro, Brazil. pp.718-728, ⟨10.1109/IPDPSW.2019.00120⟩. ⟨hal-02381750⟩
73 Consultations
108 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More