Supporting fault tolerance in a data-intensive computing middleware

Over the last 2-3 years, the importance of data-intensive computing has increasingly been recognized, closely coupled with the emergence and popularity of map-reduce for developing this class of applications. Besides programmability and ease of parallelization, fault tolerance is clearly important for data-intensive applications, because of their long running nature, and because of the potential for using a large number of nodes for processing massive amounts of data. Fault-tolerance has been an important attribute of map-reduce as well in its Hadoop implementation, where it is based on replication of data in the file system. Two important goals in supporting fault-tolerance are low overheads and efficient recovery. With these goals, this paper describes a different approach for enabling data-intensive computing with fault-tolerance. Our approach is based on an API for developing data-intensive computations that is a variation of map-reduce, and it involves an explicit programmer-decl...

• Type: Journal
• Year: 2010
• Where: IPPS
• Authors: Tekin Bicer, Wei Jiang, Gagan Agrawal
• Keywords: Data-intensive Applications, Distributed And Parallel Computing, Efficient Fault-tolerance Support, Fault-tolerance, IPPS 2010