A Lightweight Continuous Jobs Mechanism for MapReduce Frameworks - Archive ouverte HAL Access content directly
Conference Papers Year : 2013

A Lightweight Continuous Jobs Mechanism for MapReduce Frameworks


MapReduce is a programming model which allows the processing of vast amounts of data in parallel, on a large number of machines. It is particularly well suited to static or slow changing set of data since the execution time of a job is usually high. However, in practice data-centers collect data at fast rates which makes it very difficult to maintain up-to-date results. To address this challenge, we propose in this paper a generic mechanism for dealing with dynamic data in MapReduce frameworks. Long-standing MapReduce jobs, called continuous Jobs, are automatically re-executed to process new incoming data at a minimum cost. We present a simple and clean API which integrates nicely with the standard MapReduce model. Furthermore, we describe cHadoop, an implementation of our approach based on Hadoop which does not require modifications to the source code of the original framework. Thus, cHadoop can quickly be ported to any new version of Hadoop. We evaluate our proposal with two standard MapReduce applications (WordCount and WordCount-N-Count), and one real world application (RDF Query) on real datasets. Our evaluations on clusters ranging from 5 to 40 nodes demonstrate the benefit of our approach in terms of execution time and ease of use.
Fichier principal
Vignette du fichier
main.pdf (387.25 Ko) Télécharger le fichier
Origin : Files produced by the author(s)

Dates and versions

hal-00916103 , version 1 (12-12-2013)


  • HAL Id : hal-00916103 , version 1


Trong-Tuan Vu, Fabrice Huet. A Lightweight Continuous Jobs Mechanism for MapReduce Frameworks. 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2013, Jun 2013, Netherlands. pp.269-276. ⟨hal-00916103⟩
317 View
415 Download


Gmail Facebook Twitter LinkedIn More