Towards optimising distributed data streaming graphs using parallel streams

Modern scientific collaborations have opened up the opportunity of solving complex problems that involve multidisciplinary expertise and large-scale computational experiments. These experiments usually involve large amounts of data that are located in distributed data repositories running various software systems, and managed by different organisations. A common strategy to make the experiments more manageable is executing the processing steps as a workflow. In this paper, we look into the implementation of fine-grained data-flow between computational elements in a scientific workflow as streams. We model the distributed computation as a directed acyclic graph where the nodes represent the processing elements that incrementally implement specific subtasks. The processing elements are connected in a pipelined streaming manner, which allows task executions to overlap. We further optimise the execution by splitting pipelines across processes and by introducing extra parallel streams. We ...

• Type: Conference
• Year: 2010
• Where: HPDC
• Authors: Chee Sun Liew, Malcolm P. Atkinson, Jano I. van Hemert, Liangxiu Han
• Keywords: Distributed And Parallel Computing, HPDC 2010, Parallel Stream, Processing Elements, Scientific Workflows