Workload characterization for operator-based distributed stream processing applications

Operator-based programming languages provide an effective development model for large scale stream processing applications. A stream processing application consists of many runtime deployable software processing elements (PE) that work in flows to process incoming messages. Operators (OP) are logical building blocks hosted by PEs. One or more OPs can be fused into a PE at compile-time. Performance optimization for our streaming system includes compile-time fusion optimization and runtime PE-to-host deployment. One of the goals of an optimized stream application is to use minimal computing resource to sustain maximal message throughput. Characterizing the resource usage of PEs is critical for performance optimization. During compile-time optimization, OP-level resource usage is used to predict the resource usage of fused PEs. When starting an application, PE-level resource usage is used as an initial estimation by the scheduler. In this paper, we propose an efficient workload character...

• Type: Conference
• Year: 2010
• Where: DEBS
• Authors: Xiaolan J. Zhang, Sujay Parekh, Bugra Gedik, Henrique Andrade, Kun-Lung Wu
• Keywords: DEBS 2010, Distributed And Parallel Computing, OP-level Resource Usage, PE-level Resource Usage, Resource Usage