Scientific Data Repositories: Designing for a Moving Target

Managing scientific data warehouses requires constant adaptations to cope with changes in processing algorithms, computing environments, database schemas, and usage patterns. We have faced this challenge in the RHESSI Experimental Data Center (HEDC), a datacenter for the RHESSI NASA spacecraft. In this paper we describe our experience in developing HEDC and discuss in detail the design choices made. To successfully accommodate typical adaptations encountered in scientific data management systems, HEDC (i) clearly separates generic from domain specific code in all tiers, (ii) uses a file system for the actual data in combination with a DBMS to manage the corresponding meta data, and (iii) revolves around a middle tier designed to scale if more browsing or processing power is required. These design choices are valuable contributions as they address common concerns in a wide range of scientific data management systems.

• Type: Conference
• Year: 2003
• Where: SIGMOD
• Authors: Etzard Stolte, Christoph von Praun, Gustavo Alonso, Thomas R. Gross
• Keywords: Data Management Systems, Database, Scientific Data Management, Scientific Data Warehouses, SIGMOD 2003