Mission Possible: Unify HPC and Big Data Stacks Towards Application-Defined Blobs at the Storage Layer - Archive ouverte HAL Accéder directement au contenu
Article Dans Une Revue Future Generation Computer Systems Année : 2020

Mission Possible: Unify HPC and Big Data Stacks Towards Application-Defined Blobs at the Storage Layer

Résumé

HPC and Big Data stacks are completely separated today. The storage layer offers opportunities for convergence, as the challenges associated with HPC and Big Data storage are similar: trading versatility for performance. This motivates a global move towards dropping file-based, POSIX-IO compliance systems. However, on HPC platforms this is made difficult by the centralized storage architecture using file-based storage. In this paper we advocate that the growing trend of equipping HPC compute nodes with local storage redistributes the cards by enabling object storage to be deployed alongside the application on the compute nodes. Such integration of application and storage not only allows fine-grained configuration of the storage system, but also improves application portability across platforms. In addition, the single-user nature of such application-specific storage obviates the need for resource-consuming storage features like permissions or file hierarchies offered by traditional file systems. In this article we propose and evaluate Blobs (Binary Large Objects) as an alternative to distributed file systems. We factually demonstrate that it offers drop-in compatibility with a variety of existing applications while improving storage throughput by up to 28%.
Fichier principal
Vignette du fichier
HPC_BD_Convergence__Elsevier_-2.pdf (167.41 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01892682 , version 1 (10-10-2018)

Identifiants

Citer

Pierre Matri, Yevhen Alforov, Alvaro Brandon, María Pérez, Alexandru Costan, et al.. Mission Possible: Unify HPC and Big Data Stacks Towards Application-Defined Blobs at the Storage Layer. Future Generation Computer Systems, 2020, 109, pp.668-677. ⟨10.1016/j.future.2018.07.035⟩. ⟨hal-01892682⟩
485 Consultations
265 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More