Meta-data Management System for High-Performance Large-Scale Scientific Data Access

Many scientific applications manipulate large amount of data and, therefore, are parallelized on high-performance computing systems to take advantage of their computational power and memory space. The size of data processed by these large-scale applications can easily overwhelm the disk capacity of most systems. Thus, tertiary storage devices are used to store the data. The parallelization of this type of applications requires understanding of not only the data partition pattern among multiple processors but also the underlying storage architectures and the data storage pattern. In this paper, we present a meta-data management system which uses a database to record the information of datasets and manage these meta data to provide suitable I/O interface. As a result, users specify dataset names instead of data physical location to access data using optimal I/O calls without knowing the underlying storage structure. We use an astrophysics application to demonstrate that the management sy...

• Type: Conference
• Year: 2000
• Where: HIPC
• Authors: Wei-keng Liao, Xiaohui Shen, Alok N. Choudhary
• Keywords: Data Partition Pattern, Data Storage Pattern, Distributed And Parallel Computing, HIPC 2000, Tertiary Storage Devices