MOON: MapReduce On Opportunistic eNvironments

MapReduce offers a flexible programming model for processing and generating large data sets on dedicated resources, where only a small fraction of such resources are every unavailable at any given time. In contrast, when MapReduce is run on volunteer computing systems, which opportunistically harness idle desktop computers via frameworks like Condor, it results in poor performance due to the volatility of the resources, in particular, the high rate of node unavailability. Specifically, the data and task replication scheme adopted by existing MapReduce implementations is woefully inadequate for resources with high unavailability. To address this, we propose MOON, short for MapReduce On Opportunistic eNvironments. MOON extends Hadoop, an open-source implementation of MapReduce, with adaptive task and data scheduling algorithms in order to offer reliable MapReduce services on a hybrid resource architecture, where volunteer computing systems are supplemented by a small set of dedicated nod...

• Type: Conference
• Year: 2010
• Where: HPDC
• Authors: Heshan Lin, Xiaosong Ma, Jeremy S. Archuleta, Wu-chun Feng, Mark K. Gardner, Zhe Zhang
• Keywords: Adaptive Task, Data Scheduling Algorithms, Distributed And Parallel Computing, HPDC 2010, Volunteer Computing Systems