Estimating the Rate of Web Page Updates

Estimating the rate of Web page updates helps in improving the Web crawler’s scheduling policy. But, most of the Web sources are autonomous and updated independently. Clients like Web crawlers are not aware of when and how often the sources change. Unlike other studies, the process of Web page updates is modeled as non-homogeneous Poisson process and focus on determining localized rate of updates. Then various rate estimators are discussed, showing experimentally how precise they are. This paper explores two classes of problems. Firstly the localized rate of updates is estimated by dividing the given sequence of independent and inconsistent update points into consistent windows. From various experimental comparisons, the proposed Weibull estimator outperforms Duane plot(another proposed estimator) and other estimators proposed by Cho et la. and Norman Mat

• Type: Conference
• Year: 2007
• Where: IJCAI
• Authors: Sanasam Ranbir Singh
• Keywords: Artificial Intelligence, IJCAI 2007, Localized Rate, Web Crawlers, Web Page Updates