Multi-Replication with Intelligent Staging in Data-Intensive Grid Applications

Existing data grid scheduling systems handle huge data I/O via replica location services coupled with simple staging, decoupled from scheduling of computing tasks. However, when the application/workflow scales, we observe considerable degradations in performance, compared to processing within a tightly-coupled cluster. For example, when numerous nodes access the same set of files simultaneously, major performance degradation occurs even if replicas are used, due to bottlenecks that manifest in the infrastructure. Instead of resorting to expensive solutions such as parallel file systems, we propose alleviating the situation by tightly coupling replica and data transfer management with computation scheduling. In particular we propose three techniques: (1) dynamic aggregation and O(1) replication of data-staging requests across multiple nodes using a multi-replication framework, (2) replica-centric scheduling -data re-use and time-to-replication as compute scheduling metrics on the grid a...

• Type: Journal
• Year: 2006
• Where: GRID
• Authors: Yuya Machida, Shin'ichiro Takizawa, Hidemoto Nakada, Satoshi Matsuoka
• Keywords: Distributed And Parallel Computing, Grid, GRID 2006, Major Performance Degradation, Replica Location Services