Mining Console Logs for Large-Scale System Problem Detection

The console logs generated by an application contain messages that the application developers believed would be useful in debugging or monitoring the application. Despite the ubiquity and large size of these logs, they are rarely exploited in a systematic way for monitoring and debugging because they are not readily machineparsable. In this paper, we propose a novel method for mining this rich source of information. First, we combine log parsing and text mining with source code analysis to extract structure from the console logs. Second, we extract features from the structured information in order to detect anomalous patterns in the logs using Principal Component Analysis (PCA). Finally, we use a decision tree to distill the results of PCA-based anomaly detection to a format readily understandable by domain experts (e.g. system operators) who need not be familiar with the anomaly detection algorithms. As a case study, we distill over one million lines of console logs from the Hadoop f...

• Type: Conference
• Year: 2008
• Where: OSDI
• Authors: Wei Xu, Ling Huang, Armando Fox, David A. Patterson, Michael I. Jordan
• Keywords: Console Logs, Decision Tree, Operating System, OSDI 2008, Principal Component Analysis