Visual, Log-Based Causal Tracing for Performance Debugging of MapReduce Systems

Abstract—The distributed nature and large scale of MapReduce programs and systems poses two challenges in using existing profiling and debugging tools to understand MapReduce programs. Existing tools produce too much information because of the large scale of MapReduce programs, and they do not expose program behaviors in terms of Maps and Reduces. We have developed a novel non-intrusive log-analysis technique which extracts state-machine views of the control- and dataflows in MapReduce behavior from the native logs of Hadoop MapReduce systems, and it synthesizes these views to create a unified, causal view of MapReduce program behavior. This technique enables us to visualize MapReduce programs in terms of MapReduce-specific behaviors, aiding operators in reasoning about and debugging performance problems in MapReduce systems. We validate our technique and visualizations using a realworld workload, showing how to understand the structure and performance behavior of MapReduce jobs,...

• Type: Conference
• Year: 2010
• Where: ICDCS
• Authors: Jiaqi Tan, Soila Kavulya, Rajeev Gandhi, Priya Narasimhan
• Keywords: Distributed And Parallel Computing, ICDCS 2010, Large Scale, Mapreduce Programs, Program Behavior