A Comparative Experimental Study of Parallel File Systems for Large-Scale Data Processing

Large-scale scientific and business applications require data processing of ever-increasing amounts of data, fueling a demand for scalable parallel file systems comprising hundreds to thousands of disks. Modern parallel file system architectures however, span a large and complex design space. As a result, IT architects are faced with a challenge when deciding on the most appropriate parallel file system for a specific scientific or industrial application in a large-scale computing installation. Typically, the right choice depends on the characteristics of the application as well as the design assumptions built into a parallel file system. In this study, we take a close look at two prominent modern parallel file systems, PVFS2 and Lustre, and compare them experimentally on a range of benchmark-drivenscenarios modeling specific real-world applications.

• Type: Conference
• Year: 2008
• Where: USENIX
• Authors: Zoe Sebepou, Kostas Magoutis, Manolis Marazakis, Angelos Bilas
• Keywords: File Systems, Operating System, Parallel File, Parallel File Systems, USENIX 2008