Pathway: a fast and flexible unified stream data processing framework for analytical and Machine Learning applications - Archive ouverte HAL Accéder directement au contenu
Pré-Publication, Document De Travail Année : 2023

Pathway: a fast and flexible unified stream data processing framework for analytical and Machine Learning applications

Michal Bartoszkiewicz
  • Fonction : Auteur
Jan Chorowski
  • Fonction : Auteur
Adrian Kosowski
  • Fonction : Auteur
Jakub Kowalski
  • Fonction : Auteur
Sergey Kulik
  • Fonction : Auteur
Mateusz Lewandowski
  • Fonction : Auteur
Krzysztof Nowicki
  • Fonction : Auteur
Kamil Piechowiak
  • Fonction : Auteur
Olivier Ruas
  • Fonction : Auteur
Zuzanna Stamirowska
  • Fonction : Auteur
Przemyslaw Uznanski
  • Fonction : Auteur

Résumé

We present Pathway, a new unified data processing framework that can run workloads on both bounded and unbounded data streams. The framework was created with the original motivation of resolving challenges faced when analyzing and processing data from the physical economy, including streams of data generated by IoT and enterprise systems. These required rapid reaction while calling for the application of advanced computation paradigms (machinelearning-powered analytics, contextual analysis, and other elements of complex event processing). Pathway is equipped with a Table API tailored for Python and Python/SQL workflows, and is powered by a distributed incremental dataflow in Rust. We describe the system and present benchmarking results which demonstrate its capabilities in both batch and streaming contexts, where it is able to surpass state-of-the-art industry frameworks in both scenarios. We also discuss streaming use cases handled by Pathway which cannot be easily resolved with state-of-the-art industry frameworks, such as streaming iterative graph algorithms (PageRank, etc.).
Fichier principal
Vignette du fichier
pathway.pdf (1.43 Mo) Télécharger le fichier
pathway.bib (8.08 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-04159614 , version 1 (12-07-2023)

Identifiants

  • HAL Id : hal-04159614 , version 1

Citer

Michal Bartoszkiewicz, Jan Chorowski, Adrian Kosowski, Jakub Kowalski, Sergey Kulik, et al.. Pathway: a fast and flexible unified stream data processing framework for analytical and Machine Learning applications. 2023. ⟨hal-04159614⟩
143 Consultations
54 Téléchargements

Partager

Gmail Facebook X LinkedIn More