Cooperative Write-Behind Data Buffering for MPI I/O

Many large-scale production parallel programs often run for a very long time and require data checkpoint periodically to save the state of the computation for program restart and/or tracing the progress. Such a write-only pattern has become a dominant part of an application’s I/O workload and implies the importance of its optimization. Existing approaches for write-behind data buffering at both file system and MPI I/O levels have been proposed, but challenges still exist for efficient design to maintain data consistency among distributed buffers. To address this problem, we propose a buffering scheme that coordinates the compute processes to achieve the consistency control. Different from other earlier work, our design can be applied to files opened in read-write mode and handle the patterns with mixed MPI collective and independent I/O calls. Performance evaluation using BTIO and FLASH IO benchmarks is presented, which shows a significant improvement over the method without buff...

• Type: Conference
• Year: 2005
• Where: PVM
• Authors: Wei-keng Liao, Kenin Coloma, Alok N. Choudhary, Lee Ward
• Keywords: Application’s I/o Workload, MPI I/O Levels, PVM 2005, Write-behind Data Buffering