Toward Predictive Failure Management for Distributed Stream Processing Systems

Distributed stream processing systems (DSPSs) have many important applications such as sensor data analysis, network security, and business intelligence. Failure management is essential for DSPSs that often require highlyavailable system operations. In this paper, we explore a new predictive failure management approach that employs online failure prediction to achieve more efficient failure management than previous reactive or proactive failure management approaches. We employ light-weight streambased classification methods to perform online failure forecast. Based on the prediction results, the system can take differentiated failure preventions on abnormal components only. Our failure prediction model is tunable, which can achieve a desired tradeoff between failure penalty reduction and prevention cost based on a user-defined reward function. To achieve low-overhead online learning, we propose adaptive data stream sampling schemes to adaptively adjust measurement sampling rates ba...

• Type: Conference
• Year: 2008
• Where: ICDCS
• Authors: Xiaohui Gu, Spiros Papadimitriou, Philip S. Yu, Shu-Ping Chang
• Keywords: Distributed And Parallel Computing, Efficient Failure Management, Failure Management, ICDCS 2008, Predictive Failure Management