Incremental learning of system log formats

System logs come in a large and evolving variety of formats, many of which are semi-structured and/or non-standard. As a consequence, off-the-shelf tools for processing such logs often do not exist, forcing analysts to develop their own tools, which is costly and time-consuming. In this paper, we present an incremental algorithm that automatically infers the format of system log files. From the resulting format descriptions, we can generate a suite of data processing tools automatically. The system can handle large-scale data sources whose formats evolve over time. Furthermore, it allows analysts to modify inferred descriptions as desired and incorporates those changes in future revisions.

• Type: Journal
• Year: 2010
• Where: SIGOPS
• Authors: Kenny Qili Zhu, Kathleen Fisher, David Walker
• Keywords: Large-scale Data Sources, Off-the-shelf Tools, SIGOPS 2010, Tools