A Segment-Level Adaptive Data Layout Scheme for Improved Load Balance in Parallel File Systems

Abstract—Parallel file systems are designed to mask the everincreasing gap between CPU and disk speeds via parallel I/O processing. While they have become an indispensable component of modern high-end computing systems, their inadequate performance is a critical issue facing the HPC community today. Conventionally, a parallel file system stripes a file across multiple file servers with a fixed stripe size. The stripe size is a vital performance parameter, but the optimal value for it is often application dependent. How to determine the optimal stripe size is a difficult research problem. Based on the observation that many applications have different data-access clusters in one file, with each cluster having a distinguished data access pattern, we propose in this paper a segmented data layout scheme for parallel file systems. The basic idea behind the segmented approach is to divide a file logically into segments such that an optimal stripe size can be identified for each se...

• Type: Journal
• Year: 2011
• Where: CCGRID
• Authors: Huaiming Song, Yanlong Yin, Xian-He Sun, Rajeev Thakur, Samuel Lang
• Keywords: CCGRID 2011, Distributed And Parallel Computing, Optimal Stripe Size, Stripe Size, fixed Stripe Size