Supporting Generic Cost Models for Wide-Area Stream Processing

— Existing stream processing systems are optimized for a specific metric, which may limit their applicability to diverse applications and environments. This paper presents XFlow, a generic data stream collection, processing, and dissemination system that addresses this limitation efficiently. XFlow can express and optimize a variety of optimization metrics and constraints by distributing stream processing queries across a wide-area network. It uses metric-independent decentralized algorithms that work on localized, aggregated statistics, while avoiding local optima. To facilitate light-weight dynamic changes on the query deployment, XFlow relies on a loosely-coupled, flexible architecture consisting of multiple publish-subscribe overlay trees that can gracefully scale and adapt to changes to network and workload conditions. Based on the desired performance goals, the system progressively refines the query deployment, the structure of the overlay trees, as well as the statistics c...

• Type: Conference
• Year: 2009
• Where: ICDE
• Authors: Olga Papaemmanouil, Ugur Çetintemel, John Jannotti
• Keywords: Database, ICDE 2009, Query Deployment, Stream Processing, Stream Processing Systems