An Efficient MPI-IO for Noncontiguous Data Access over InfiniBand

Noncontiguous data access is a very common access pattern in many scientific applications. Using POSIX I/O to access many pieces of noncontiguous data segments will generate a lot of I/O requests that cause the I/O system perform poorly. Tow-phase I/O, also called collective I/O, applied by MPI-IO provides a good method to optimize noncontiguous I/O operations. However, this method requires many data segments being redistributed. This redistribution needs some information been calculated in advance and lots of data segments are transmitted via network more than once. With I/O-intensive applications, the aggregate size of these data segments being redistributed becomes significant large. This additional overhead will degrade the system performance. In this thesis, we extend the collective I/O method and propose a new I/O scheme to avoid retransmission of data segments by applying RDMA gather/scatter operations supported by InfiniBand hardware. We also extend the “view” and “datat...

• Type: Conference
• Year: 2005
• Where: ISPAN
• Authors: Ding-Yong Hong, Ching-Wen You, Yeh-Ching Chung
• Keywords: Collective I/o, Data Segments, Distributed And Parallel Computing, ISPAN 2005, Noncontiguous I/o Operations