A Hadoop MapReduce Performance Prediction Method - Archive ouverte HAL
Communication Dans Un Congrès Année : 2013

A Hadoop MapReduce Performance Prediction Method

Résumé

More and more Internet companies rely on large scale data analysis as part of their core services for tasks such as log analysis, feature extraction or data filtering. Map-Reduce, through its Hadoop implementation, has proved to be an efficient model for dealing with such data. One important challenge when performing such analysis is to predict the performance of individual jobs. In this paper, we propose a simple framework to predict the performance of Hadoop jobs. It is composed of a dynamic light-weight Hadoop job analyzer, and a prediction module using locally weighted regression methods. Our framework makes some theoretical cost models more practical, and also well fits for the diversification of the jobs and clusters. It can also help those users who want to predict the cost when applying for an on- demand cloud service. At the end, we do some experiments to verify our framework.
Fichier principal
Vignette du fichier
HadoopPrediction.pdf (925.83 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)
Loading...

Dates et versions

hal-00918329 , version 1 (21-01-2014)

Identifiants

Citer

Ge Song, Zide Meng, Fabrice Huet, Frederic Magoules, Lei Yu, et al.. A Hadoop MapReduce Performance Prediction Method. HPCC 2013, Nov 2013, Zhangjiajie, China. pp.820-825, ⟨10.1109/HPCC.and.EUC.2013.118⟩. ⟨hal-00918329⟩
660 Consultations
1975 Téléchargements

Altmetric

Partager

More