SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence - Archive ouverte HAL Accéder directement au contenu
Communication Dans Un Congrès Année : 2018

SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence

Résumé

Cloud developers traditionally rely on purpose-specific services to provide the storage model they need for an application. In contrast, HPC developers have a much more limited choice, typically restricted to a centralized parallel file system for persistent storage. Unfortunately, these systems often offer low performance when subject to highly concurrent, conflicting I/O patterns. This makes difficult the implementation of inherently concurrent data structures such as distributed shared logs. Yet, this data structure is key to applications such as computational steering, data collection from physical sensor grids, or discrete event generators. In this paper we tackle this issue. We present SLoG, shared log middleware providing a shared log abstraction over a parallel file system, designed to circumvent the aforementioned limitations. We evaluate SLoG's design on up to 100,000 cores of the Theta supercomputer: the results show high append velocity at scale while also providing substantial benefits for other persistent backend storage systems.
Fichier principal
Vignette du fichier
HPC_Logging___ICDCS_18_Short_Paper.pdf (190.78 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-01892685 , version 1 (10-10-2018)

Identifiants

Citer

Pierre Matri, Philip Carns, Robert Ross, Alexandru Costan, María Pérez, et al.. SLoG: Large-Scale Logging Middleware for HPC and Big Data Convergence. ICDCS 2018 - IEEE 38th International Conference on Distributed Computing Systems, Jul 2018, Vienna, Austria. pp.1-6, ⟨10.1109/ICDCS.2018.00156⟩. ⟨hal-01892685⟩
451 Consultations
279 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More