Focusing on novelty: a crawling strategy to build diverse language models

Word prediction performed by language models has an important role in many tasks as e.g. word sense disambiguation, speech recognition, hand-writing recognition, query spelling and query segmentation. Recent research has exploited the textual content of the Web to create language models. In this paper, we propose a new focused crawling strategy to collect Web pages that focuses on novelty in order to create diverse language models. In each crawling cycle, the crawler tries to fill the gaps present in the current language model built from previous cycles, by avoiding visiting pages whose vocabulary is already well represented in the model. It relies on an information theoretic measure to identify these gaps and then learns link patterns to pages in these regions in order to guide its visitation policy. To handle constantly evolving domains, a key feature of our crawler approach is its ability to adjust its focus as the crawl progresses. We evaluate our approach in two different scena...

• Type: Journal
• Year: 2011
• Where: CIKM
• Authors: Luciano Barbosa, Srinivas Bangalore
• Keywords: CIKM 2011, Information Technology, Language Models, Link Patterns, Word Sense Disambiguation