Co-Scheduling of Computation and Data on Computer Clusters

Scientific investigations have to deal with rapidly growing amounts of data from simulations and experiments. During data analysis, scientists typically want to extract subsets of the data and perform computations on them. In order to speed up the analysis, computations are performed on distributed systems such as computer clusters, or Grid systems. A well-known difficult problem is to build systems that execute the computations and data movement in a coordinated fashion. In this paper, we describe an architecture for executing co-scheduled tasks of computation and data movement on a computer cluster that takes advantage of two technologies currently being used in distributed Grid systems. The first is Condor, that manages the scheduling and execution of distributed computation, and the second is Storage Resource Managers (SRMs) that manage the space usage and content of storage systems. This is achieved by including the information about the availability of files on the nodes pro...

• Type: Conference
• Year: 2005
• Where: SSDBM
• Authors: Alexandru Romosan, Doron Rotem, Arie Shoshani, Derek Wright
• Keywords: Computation, Computer Cluster, Database, Grid Systems, SSDBM 2005