Lossless compression for large scale cluster logs

The growing computational and storage needs of several scientific applications mandate the deployment of extreme-scale parallel machines, such as IBM’s Blue Gene/L which can accommodate as many as 128K processors. One of the biggest challenges these systems face, is to manage generated system logs while deploying in production environments. Large amount of log data is created over extended period of time, across thousands of processors. These logs generated can be voluminous because of the large temporal and spatial dimensions, and containing records which are repeatedly entered to the log archive. Storing and transferring such large amount of log data is a challenging problem. Commonly used generic compression utilities are not optimal for such large amount of data considering a number of performance requirements. In this paper we propose a compression algorithm which preprocesses these logs before trying out any standard compression utilities. The compression ratios and times for ...

• Type: Conference
• Year: 2006
• Where: IPPS
• Authors: R. Balakrishnan, Ramendra K. Sahoo
• Keywords: Compression Utilities, Distributed And Parallel Computing, Generic Compression Utilities, IPPS 2006, Log Data