What's new on the web?: the evolution of the web from a search engine perspective

We seek to gain improved insight into how Web search engines should cope with the evolving Web, in an attempt to provide users with the most up-to-date results possible. For this purpose we collected weekly snapshots of some 150 Web sites over the course of one year, and measured the evolution of content and link structure. Our measurements focus on aspects of potential interest to search engine designers: the evolution of link structure over time, the rate of creation of new pages and new distinct content on the Web, and the rate of change of the content of existing pages under search-centric measures of degree of change. Our findings indicate a rapid turnover rate of Web pages, i.e., high rates of birth and death, coupled with an even higher rate of turnover in the hyperlinks that connect them. For pages that persist over time we found that, perhaps surprisingly, the degree of content shift as measured using TF.IDF cosine distance does not appear to be consistently correlated with t...

• Type: Conference
• Year: 2004
• Where: WWW
• Authors: Alexandros Ntoulas, Junghoo Cho, Christopher Olston
• Keywords: Experience Little Change, Internet Technology, Rapid Turnover Rate, Web Search Engines, WWW 2004