Detouring and replication for fast and reliable internet-scale stream processing

iFlow is a replication-based system that can achieve both fast and reliable processing of high volume data streams on the Internet scale. iFlow uses a low degree of replication in conjunction with detouring techniques to overcome network congestion and outages. Computation over iFlow can be expressed as a graph of operators. To cope with varying system conditions these operators continually migrate in a manner that improves performance and availability at the same time. In this paper, we first provide an overview of our iFlow system. Next, we detail how our detouring technique works in the face of network failures to provide high availability for time critical applications. The paper also includes a description of our implementation and preliminary evaluation results demonstrating that iFlow outperforms previous solutions with less overhead. Finally, the paper concludes with our plans for enhancing replication and detouring capabilities. Categories and Subject Descriptors C.4 [Perform...

• Type: Conference
• Year: 2010
• Where: HPDC
• Authors: Christopher McConnell, Fan Ping, Jeong-Hyon Hwang
• Keywords: Distributed And Parallel Computing, HPDC 2010, IFlow System, Reliability Keywords Availability, Volume Data Streams