Enabling high-speed asynchronous data extraction and transfer using DART

As the complexity and scale of current scientific and engineering applications grow, managing and transporting the large amounts of data they generate is quickly becoming a significant challenge. The increasing application runtimes and the high cost of high performance computing resources make online data extraction and analysis a key requirement in addition to traditional data I/O and archiving. To be effective, online data extraction and transfer should impose minimal additional synchronization requirements, should have minimal impact on the computational performance, maintain overall Quality of Service, and ensure that no data is lost. In this paper, we present DART (Decoupled and Asynchronous Remote Transfers), an efficient data transfer substrate that effectively addresses these requirements. DART is a thin software layer built on RDMA technology to enable fast, low-overhead and asynchronous access to data from a running simulation, and support high-throughput, low-latency data t...

• Type: Journal
• Year: 2010
• Where: CONCURRENCY
• Authors: Ciprian Docan, Manish Parashar, Scott Klasky
• Keywords: CONCURRENCY 2010, DART, Data Extraction, Simulation