Improving Web search efficiency via a locality based static pruning method

The unarguably fast, and continuous, growth of the volume of indexed (and indexable) documents on the Web poses a great challenge for search engines. This is true regarding not only search effectiveness but also time and space efficiency. In this paper we present an index pruning technique targeted for search engines that addresses the latter issue without disconsidering the former. To this effect, we adopt a new pruning strategy capable of greatly reducing the size of search engine indices. Experiments using a real search engine show that our technique can reduce the indices' storage costs by up to 60% over traditional lossless compression methods, while keeping the loss in retrieval precision to a minimum. When compared to the indices size with no compression at all, the compression rate is higher than 88%, i.e., less than one eighth of the original size. More importantly, our results indicate that, due to the reduction in storage overhead, query processing time can be reduced ...

• Type: Conference
• Year: 2005
• Where: WWW
• Authors: Edleno Silva de Moura, Célia Francisca dos Santos, Daniel R. Fernandes, Altigran Soares da Silva, Pável Calado, Mario A. Nascimento
• Keywords: Internet Technology, Real Search Engine, Search Engine Indices, Search Engines, WWW 2005