Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis

Abstract -- Detection of execution anomalies is very important for the maintenance, development, and performance refinement of large scale distributed systems. Execution anomalies include both work flow errors and low performance problems. People often use system logs produced by distributed systems for troubleshooting and problem diagnosis. However, manually inspecting system logs to detect anomalies is unfeasible due to the increasing scale and complexity of distributed systems. Therefore, there is a great demand for automatic anomaly detection techniques based on log analysis. In this paper, we propose an unstructured log analysis technique for anomaly detection. In the technique, we propose a novel algorithm to convert free form text messages in log files to log keys without heavily relying on application specific knowledge. The log keys correspond to the log-print statements in the source code which can provide cues of system execution behavior. After converting log messages to lo...

• Type: Conference
• Year: 2009
• Where: ICDM
• Authors: Qiang Fu, Jian-Guang Lou, Yi Wang, Jiang Li
• Keywords: Data Mining, Distributed System, Execution Anomalies, ICDM 2009, Log Analysis