Towards automatic optimization of MapReduce programs

Timely and cost-effective processing of large datasets has become a critical ingredient for the success of many academic, government, and industrial organizations. The combination of MapReduce frameworks and cloud computing is an attractive proposition for these organizations. However, even to run a single program in a MapReduce framework, a number of tuning parameters have to set by users or system administrators. Users often run into performance problems because they don’t know how to set these parameters, or because they don’t even know that these parameters exist. With MapReduce being a relatively new technology, it is not easy to find qualified administrators. In this position paper, we make a case for techniques to automate the setting of tuning parameters for MapReduce programs. The objective is to provide good outof-the-box performance for ad hoc MapReduce programs run on large datasets. This feature can go a long way towards improving the productivity of users who lack ...

• Type: Conference
• Year: 2010
• Where: CLOUD
• Authors: Shivnath Babu
• Keywords: CLOUD 2010, Distributed And Parallel Computing, Large Datasets, MapReduce Framework, Parameters