Improving RDMA-based MPI eager protocol for frequently-used buffers

MPI is the main standard for communication in high-performance clusters. MPI implementations use the Eager protocol to transfer small messages. To avoid the cost of memory registration and prenegotiation, the Eager protocol involves a data copy to intermediate buffers at both sender and receiver sides. In this paper, however, we propose that when a user buffer is used frequently in an application, it is more efficient to register the sender buffer and avoid the sender-side data copy. The performance results of our proposed Eager protocol on MVAPICH2 over InfiniBand indicate that up to 14% improvement can be achieved in a single medium-size message latency, comparable to a maximum 15% theoretical improvement on our platform. We also show that collective communications such as broadcast can benefit from the new protocol by up to 19%. In addition, the communication time in MPI applications with high buffer reuse is improved using this technique.

• Type: Conference
• Year: 2009
• Where: IPPS
• Authors: Mohammad J. Rashti, Ahmad Afsahi
• Keywords: Data Copy, Distributed And Parallel Computing, Eager Protocol, IPPS 2009, MPI Implementations