Sciweavers

HPDC
2010
IEEE

Towards optimising distributed data streaming graphs using parallel streams

14 years 1 months ago
Towards optimising distributed data streaming graphs using parallel streams
Modern scientific collaborations have opened up the opportunity of solving complex problems that involve multidisciplinary expertise and large-scale computational experiments. These experiments usually involve large amounts of data that are located in distributed data repositories running various software systems, and managed by different organisations. A common strategy to make the experiments more manageable is executing the processing steps as a workflow. In this paper, we look into the implementation of fine-grained data-flow between computational elements in a scientific workflow as streams. We model the distributed computation as a directed acyclic graph where the nodes represent the processing elements that incrementally implement specific subtasks. The processing elements are connected in a pipelined streaming manner, which allows task executions to overlap. We further optimise the execution by splitting pipelines across processes and by introducing extra parallel streams. We ...
Chee Sun Liew, Malcolm P. Atkinson, Jano I. van He
Added 09 Nov 2010
Updated 09 Nov 2010
Type Conference
Year 2010
Where HPDC
Authors Chee Sun Liew, Malcolm P. Atkinson, Jano I. van Hemert, Liangxiu Han
Comments (0)