Data management for large-scale scientific computations in high performance distributed systems

With the increasing number of scientific applications manipulating huge amounts of data, effective high-level data management is an increasingly important problem. Unfortunately, so far the solutions to the high-level data management problem either require deep understanding of specific storage architectures and file layouts (as in high-performance file storage systems) or produce unsatisfactory I/O performance in exchange for ease-of-use and portability (as in relational DBMSs). In this paper we present a novel application development environment which is built around an active meta-data management system (MDMS) to handle high-level data in an effective manner. The key components of our three-tiered architecture are user application, the MDMS, and a hierarchical storage system (HSS). Our environment overcomes the performance problems of pure database-oriented solutions, while maintaining their advantages in terms of ease-of-use and portability. The high levels of performance are achi...

• Type: Journal
• Year: 2000
• Where: CLUSTER
• Authors: Alok N. Choudhary, Mahmut T. Kandemir, Jaechun No, Gokhan Memik, Xiaohui Shen, Wei-keng Liao, Harsha S. Nagesh, Sachin More, Valerie E. Taylor, Rajeev Thakur, Rick L. Stevens
• Keywords: Application Development Environment, CLUSTER 2000, Data Management Problem, Distributed And Parallel Computing, Specific Storage Architectures