Diarchy: An Optimized Management Approach for MapReduce Masters - Archive ouverte HAL Access content directly
Conference Papers Year : 2015

Diarchy: An Optimized Management Approach for MapReduce Masters


The MapReduce community is progressively replacing the classic Hadoop with Yarn, the second-generation Hadoop (MapReduce 2.0). This transition is being made due to many reasons, but primarily because of some scalability drawbacks of the classic Hadoop. The new framework has appropriately addressed this issue and is being praised for its multi-functionality. In this paper we carry out a probabilistic analysis that emphasizes some reliability concerns of Yarn at the job master level. This is a critical point, since the failures of a job master involves the failure of all the workers managed by such a master. In this paper, we propose Diarchy, a novel system for the management of job masters. Its aim is to increase the reliability of Yarn, based on the sharing and backup of responsibilities between two masters working as peers. The evaluation results show that Diarchy outperforms the reliability performance of Yarn in different setups, regardless of cluster size, type of job, or average failure rate and suggest a positive impact of this approach compared to the traditional, single-master Hadoop architecture.

Dates and versions

hal-01249151 , version 1 (30-12-2015)



Bunjamin Memishi, María S. Pérez-Hernández, Gabriel Antoniu. Diarchy: An Optimized Management Approach for MapReduce Masters. ICCS 2015: Proceedings of the International Conference on Computational Science, Computational Science at the Gates of Nature, Jun 2015, Reykjavík, Iceland. pp.9--18, ⟨10.1016/j.procs.2015.05.179⟩. ⟨hal-01249151⟩
210 View
0 Download



Gmail Facebook X LinkedIn More