Small-file access in parallel file systems

Abstract—Today’s computational science demands have resulted in ever larger parallel computers, and storage systems have grown to match these demands. Parallel file systems used in this environment are increasingly specialized to extract the highest possible performance for large I/O operations, at the expense of other potential workloads. While some applications have adapted to I/O best practices and can obtain good performance on these systems, the natural I/O patterns of many applications result in generation of many small files. These applications are not well served by current parallel file systems at very large scale. This paper describes five techniques for optimizing smallfile access in parallel file systems for very large scale systems. These five techniques are all implemented in a single parallel file system (PVFS) and then systematically assessed on two test platforms. A microbenchmark and the mdtest benchmark are used to evaluate the optimizations at an unprece...

• Type: Conference
• Year: 2009
• Where: IPPS
• Authors: Philip H. Carns, Samuel Lang, Robert B. Ross, Murali Vilayannur, Julian M. Kunkel, Thomas Ludwig 0002
• Keywords: Distributed And Parallel Computing, IPPS 2009, Large Scale, Larger Parallel Computers, Parallel file Systems