Improving Quality of Training Data for Learning to Rank Using Click-Through Data

In information retrieval, relevance of documents with respect to queries is usually judged by humans, and used in evaluation and/or learning of ranking functions. Previous work has shown that certain level of noise in relevance judgments has little effect on evaluation, especially for comparison purposes. Recently learning to rank has become one of the major means to create ranking models in which the models are automatically learned from the data derived from a large number of relevance judgments. As far as we know, there was no previous work about quality of training data for learning to rank, and this paper tries to study the issue. Specifically, we address three problems. Firstly, we show that the quality of training data labeled by humans has critical impact on the performance of learning to rank algorithms. Secondly, we propose detecting relevance judgment errors using click-through data accumulated at a search engine. Two discriminative models, referred to as sequential depende...

• Type: Conference
• Year: 2010
• Where: WSDM
• Authors: Jingfang Xu, Chuanliang Chen, Gu Xu, Hang Li, Elbio Renato Torres Abib
• Keywords: Data Mining, Evaluation And/or Learning, Relevance Judgment Errors, Relevance Judgments, WSDM 2010