Compressed collections for simulated crawling

Collections are a fundamental tool for reproducible evaluation of information retrieval techniques. We describe a new method for distributing the document lengths and term counts (a.k.a. within-document frequencies) of a web snapshot in a highly compressed and nonetheless quickly accessible form. Our main application is reproducibility of the behaviour of focused crawlers: by coupling our collection with the corresponding web graph compressed with WebGraph [3] we make it possible to apply text-based machine learning tools to the collection, while keeping the data set footprint small. We describe a collection based on a crawl of 100 Mpages of the .uk domain, publicly available in bundle with a Java open-source implementation of our techniques.

• Type: Journal
• Year: 2008
• Where: SIGIR
• Authors: Alessio Orlandi, Sebastiano Vigna
• Keywords: Corresponding Web Graph, Fundamental Tool, Information Retrieval Techniques, Information Technology, SIGIR 2008