Phoenix rebirth: Scalable MapReduce on a large-scale shared-memory system

Abstract—Dynamic runtimes can simplify parallel programming by automatically managing concurrency and locality without further burdening the programmer. Nevertheless, implementing such runtime systems for large-scale, shared-memory systems can be challenging. This work optimizes Phoenix, a MapReduce runtime for shared-memory multi-cores and multiprocessors, on a quad-chip, 32-core, 256-thread UltraSPARC T2+ system with NUMA characteristics. We show how a multi-layered approach that comprises optimizations on the algorithm, implementation, and OS interaction leads to significant speedup improvements with 256 threads (average of 2.5× higher speedup, maximum of 19×). We also identify the roadblocks that limit the scalability of parallel runtimes on shared-memory systems, which are inherently tied to the OS scalability on large-scale systems.

• Type: Conference
• Year: 2009
• Where: IISWC
• Authors: Richard M. Yoo, Anthony Romano, Christos Kozyrakis
• Keywords: Abstract—Dynamic Runtimes, IISWC 2009, Operating Systems, Shared-memory Multi-cores, Shared-memory Systems