HDF5-FastQuery: Accelerating Complex Queries on HDF Datasets using Fast Bitmap Indices

Large scale scientific data is often stored in scientific data formats such as FITS, netCDF and HDF. These storage formats are of particular interest to the scientific user community since they provide multi-dimensional storage and retrieval. However, one of the drawbacks of these storage formats is that they do not support semantic indexing which is important for interactive data analysis where scientists look for features of interests such as “Find all supernova explosions where energy > 105 and temperature > 106 ”. In this paper we present a novel approach called HDF5FastQuery to accelerate the data access of large HDF5 files by introducing multi-dimensional semantic indexing. Our implementation leverages an efficient indexing technology called bitmap indexing that has been widely used in the database community. Bitmap indices are especially well suited for interactive exploration of large-scale readonly data. Storing the bitmap indices into the HDF5 file has the fo...

• Type: Conference
• Year: 2006
• Where: SSDBM
• Authors: Luke J. Gosink, John Shalf, Kurt Stockinger, Kesheng Wu, Wes Bethel
• Keywords: Database, Multi-dimensional Semantic Indexing, Scientific Data, Semantic Indexing, SSDBM 2006