Evaluating Search Engines by Modeling the Relationship Between Relevance and Clicks

We propose a model that leverages the millions of clicks received by web search engines to predict document relevance. This allows the comparison of ranking functions when clicks are available but complete relevance judgments are not. After an initial training phase using a set of relevance judgments paired with click data, we show that our model can predict the relevance score of documents that have not been judged. These predictions can be used to evaluate the performance of a search engine, using our novel formalization of the confidence of the standard evaluation metric discounted cumulative gain (DCG), so comparisons can be made across time and datasets. This contrasts with previous methods which can provide only pair-wise relevance judgments between results shown for the same query. When no relevance judgments are available, we can identify the better of two ranked lists up to 82% of the time, and with only two relevance judgments for each query, we can identify the better rank...

• Type: Conference
• Year: 2007
• Where: NIPS
• Authors: Ben Carterette, Rosie Jones
• Keywords: Complete Relevance Judgments, Information Technology, NIPS 2007, Relevance Judgments, Web Search