Incremental web page template detection

Most template detection methods process web pages in batches that a newly crawled page can not be processed until enough pages have been collected. This results in large storage consumption and a huge delay of data refreshing. In this paper, we present an incremental framework to detect templates in which a page is processed as soon as it has been crawled. In this framework, we don't need to cache any web page. Experiments show that our framework consumes less than 7% storage than traditional methods. And also the speed of data refreshing is accelerated because of the incremental manner. Categories and Subject Descriptors: H.3.3 [Information Systems]: Information Search and Retrieval General Terms: Experimentation, Algorithms

• Type: Conference
• Year: 2008
• Where: WWW
• Authors: Yu Wang, Binxing Fang, Xueqi Cheng, Li Guo, Hongbo Xu
• Keywords: Data Refreshing, Incremental Framework, Internet Technology, Most Template Detection, WWW 2008