Using Subfiling to Improve Programming Flexibility and Performance of Parallel Shared-file I/O

There are two popular parallel I/O programming styles used by modern scientific computational applications: unique-file and shared-file. Unique-file I/O usually gives satisfactory performance, but its major drawback is that managing a large number of files can overwhelm the task of postsimulation data processing. Shared-file I/O produces fewer files and allows arrays partitioned among processes to be saved in the canonical order. As the number of processors on modern parallel machines increases into thousands and more, the problem size and in turn the global array size also increase proportionally. It is not practical to manage files of size each larger than a few hundreds of GB. Hence, to seek a middle ground between these two I/O styles, we propose a subfiling scheme that divides a large multi-dimensional global array into smaller subarrays, each saved in a smaller file, named subfile. Subfiling is implemented on top of MPI-IO. We also incorporate it into the parallel netCDF library ...

• Type: Journal
• Year: 2009
• Where: ICPP
• Authors: Kui Gao, Wei-keng Liao, Arifa Nisar, Alok N. Choudhary, Robert B. Ross, Robert Latham
• Keywords: Distributed And Parallel Computing, Global Array, Global Array Size, ICPP 2009, NetCDF File Header