Understanding the effects and implications of compute node related failures in hadoop

Hadoop has become a critical component in today’s cloud environment. Ensuring good performance for Hadoop is paramount for the wide-range of applications built on top of it. In this paper we analyze Hadoop’s behavior under failures involving compute nodes. We find that even a single failure can result in inflated, variable and unpredictable job running times, all undesirable properties in a distributed system. We systematically track the causes underlying this distressing behavior. First, we find that Hadoop makes unrealistic assumptions about task progress rates. These assumptions can be easily invalidated by the cloud environment and, more surprisingly, by Hadoop’s own design decisions. The result are significant inefficiencies in Hadoop’s speculative execution algorithm. Second, failures are re-discovered individually by each task at the cost of great degradation in job running time. The reason is that Hadoop focuses on extreme scalability and thus trades off possible ...

• Type: Journal
• Year: 2012
• Where: HPDC
• Authors: Florin Dinu, T. S. Eugene Ng
• Keywords: Computer Systems Organization, Distributed And Parallel Computing, Execution Algorithm, Failure Semantics, HPDC 2012