Towards an understanding of the performance of MPI-IO in Lustre file systems

—Lustre is becoming an increasingly important file system for large-scale computing clusters. The problem, however, is that many data-intensive applications use MPI-IO for their I/O requirements, and MPI-IO performs poorly in a Lustre file system environment. While this poor performance has been well documented, the reasons for such performance are currently not well understood. Our research suggests that the primary performance issues have to do with the assumptions underpinning most of the parallel I/O optimizations implemented in MPI-IO, which do not appear to hold in a Lustre environment. Perhaps the most important assumption is that optimal performance is obtained by performing large, contiguous I/O operations. However, the research results presented in this poster show that this is often the worst approach to take in a Lustre file system. In fact, we found that the best performance is often achieved when each process performs a series of smaller, noncontiguous I/O requests. In ...

• Type: Conference
• Year: 2008
• Where: CLUSTER
• Authors: Jeremy Logan, Phillip M. Dickens
• Keywords: CLUSTER 2008, Cluster Computing, File System, Lustre File, Lustre File System