Sciweavers

GLOBE
2013
Springer

Data Partitioning for Minimizing Transferred Data in MapReduce

10 years 7 months ago
Data Partitioning for Minimizing Transferred Data in MapReduce
Reducing data transfer in MapReduce’s shuffle phase is very important because it increases data locality of reduce tasks, and thus decreases the overhead of job executions. In the literature, several optimizations have been proposed to reduce data transfer between mappers and reducers. Nevertheless, all these approaches are limited by how intermediate key-value pairs are distributed over map outputs. In this paper, we address the problem of high data transfers in MapReduce, and propose a technique that repartitions tuples of the input datasets, and thereby optimizes the distribution of key-values over mappers, and increases the data locality in reduce tasks. Our approach captures the relationships between input tuples and intermediate keys by monitoring the execution of a set of MapReduce jobs which are representative of the workload. Then, based on those relationships, it assigns input tuples to the appropriate chunks. We evaluated our approach through experimentation in a Hadoop de...
Miguel Liroz-Gistau, Reza Akbarinia, Divyakant Agr
Added 28 Apr 2014
Updated 28 Apr 2014
Type Journal
Year 2013
Where GLOBE
Authors Miguel Liroz-Gistau, Reza Akbarinia, Divyakant Agrawal, Esther Pacitti, Patrick Valduriez
Comments (0)