Dynamic Scheduling of MapReduce Shuffle under Bandwidth Constraints - Archive ouverte HAL
Communication Dans Un Congrès Année : 2014

Dynamic Scheduling of MapReduce Shuffle under Bandwidth Constraints

Résumé

Whether it is for e-science or business, the amount of data produced every year is growing at a high rate. Managing and processing those data raises new challenges. MapReduce is one answer to the need for scalable tools able to handle the amount of data. It imposes a general structure of computation and let the implementation perform its optimizations. During the computation, there is a phase called shuffle where every node sends a possibly large amount of data to every other node. This paper proposes and evaluates two algorithms to improve data transfers during the shuffle phase under bandwidth constraints.

Dates et versions

hal-01194493 , version 1 (07-09-2015)

Identifiants

Citer

Sylvain Gault, Christian Pérez. Dynamic Scheduling of MapReduce Shuffle under Bandwidth Constraints. Euro-Par 2014: Parallel Processing Workshops, Aug 2014, Porto, Portugal. pp.117-128, ⟨10.1007/978-3-319-14325-5_11⟩. ⟨hal-01194493⟩
90 Consultations
0 Téléchargements

Altmetric

Partager

More