Middleware support for many-task computing

Many-task computing aims to bridge the gap between two computing paradigms, high throughput computing and high performance computing. Many-task computing denotes highperformance computations comprising multiple distinct activities, coupled via file system operations. The aggregate number of tasks, quantity of computing, and volumes of data may be extremely large. Traditional techniques found in production systems in the scientific community to support many-task computing do not scale to today’s largest systems, due to issues in local resource manager scalability and granularity, efficient utilization of the raw hardware, long wait queue times, and shared/parallel file system contention and scalability. To address these limitations, we adopted a “top-down” approach to building a middleware called Falkon, to support the most demanding manytask computing applications at the largest scales. Falkon (Fast and Light-weight tasK executiON framework) integrates (1) multi-level scheduling ...

• Type: Journal
• Year: 2010
• Where: CLUSTER
• Authors: Ioan Raicu, Ian T. Foster, Mike Wilde, Zhao Zhang, Kamil Iskra, Peter H. Beckman, Yong Zhao, Alexander S. Szalay, Alok N. Choudhary, Philip Little, et al.
• Keywords: CLUSTER 2010, Distributed And Parallel Computing, Magnitude Improvements, Micro Benchmarks, Raw Hardware