Kvasir: Empowering Hadoop with Knowledge - Archive ouverte HAL Accéder directement au contenu
Rapport (Rapport De Recherche) Année : 2015

Kvasir: Empowering Hadoop with Knowledge

Nathanaël Cheriere

Résumé

The exponential growth of collected data poses the challenge of efficient data processing among others. The MapReduce paradigm introduced by Google, has seduced the world of Big Data with its simplicity, power, and also with its open source implementation: Hadoop. While being conceived for processing batches of jobs, it became necessary to share the resources between users with a scheduler. Several built-in schedulers (i.e. FIFO, Fair, and Capacity schedulers) have been therefore introduced in Hadoop to operate shared Hadoop clusters. Most of Hadoop schedulers are based on homogeneity hypotheses about the jobs and the nodes and therefore strongly rely on the location of the input data when scheduling tasks. However, our study revealed that Hadoop is a highly dynamic environment (e.g., variation in task duration within a job and across different jobs). Even worse, clouds are multi-tenant environments which in turn introduce more heterogeneity and dynamicity in Hadoop clusters. As a result, relying on static knowledge (i.e. data location) may lead to wrong scheduling decisions. In this thesis we introduce Kvasir, a scheduling adaptive optimiser that fits between Hadoop and its scheduler. Kvasir aims to provide an up-to-date knowledge that reflects the dynamicity of the environment while being light-weight and performance-oriented. The provided information is an estimation of the duration of any task on any node, even before it is launched. To obtain such an estimation, Kvasir collects the information already provided by heartbeat messages (speed of execution) and feeds this information to a hierarchical clustering algorithm which groups together similar nodes to increase the accuracy of the predicted durations. The utility of Kvasir is demonstrated by the implementation of several schedulers that would have been hard to implemented with only Hadoop. Our experimental results show that Kvasir implementation of FIFO (i.e. KFifo) speeds up jobs by an average factor of 1.69 in a homogeneous environment and by a average factor of 2.73 for a heterogeneous environment when compared to the state-of-the-art Hadoop's FIFO scheduler.
Fichier non déposé

Dates et versions

hal-01244671 , version 1 (16-12-2015)

Identifiants

  • HAL Id : hal-01244671 , version 1

Citer

Nathanaël Cheriere. Kvasir: Empowering Hadoop with Knowledge: Exploring new scheduling techniques for processing big-data applications in Hadoop. [Research Report] Master Inria; ENS Rennes. 2015. ⟨hal-01244671⟩

Collections

LARA UNIV-RENNES
93 Consultations
0 Téléchargements

Partager

Gmail Facebook X LinkedIn More