Parallel and Distributed Astronomical Data Analysis on Grid Datafarm

A comprehensive study of the whole petabyte-scale archival data of astronomical observatories has a possibility of new science and new knowledge in the field, while it was not feasible so far due to lack of enough data analysis environment. The Grid Datafarm architecture is designed for global petabyte-scale data-intensive computing, which provides a Grid file system with file replica management for fault tolerance and load balancing, and parallel and distributed data computing support for a set of files, to meet with the requirements of the comprehensive study of the whole archival data. In the paper, we discuss about worldwide parallel and distributed data analysis in the observational astronomical field. The archival data is stored, replicated and dispersed in a Gfarm file system. All the astronomical data analysis tools successfully access files in Gfarm file system without any code modification, using a syscall hooking library regardless of file replica locations. Perfo...

• Type: Conference
• Year: 2004
• Where: GRID
• Authors: Naotaka Yamamoto, Osamu Tatebe, Satoshi Sekiguchi
• Keywords: Archival Data, Data Analysis, GRID 2004, file Replicas