Top-k learning to rank: labeling, ranking and evaluation

In this paper, we propose a novel top-k learning to rank framework, which involves labeling strategy, ranking model and evaluation measure. The motivation comes from the difficulty in obtaining reliable relevance judgments from human assessors when applying learning to rank in real search systems. The traditional absolute relevance judgment method is difficult in both gradation specification and human assessing, resulting in high level of disagreement on judgments. While the pairwise preference judgment, as a good alternative, is often criticized for increasing the complexity of judgment from O(n) to O(n log n). Considering the fact that users mainly care about top ranked search results, we propose a novel top-k labeling strategy which adopts the pairwise preference judgment to generate the top k ordering items from n documents (i.e. top-k ground-truth) in a manner similar to that of HeapSort. As a result, the complexity of judgment is reduced to O(n log k). With the topk ground-tru...

• Type: Journal
• Year: 2012
• Where: SIGIR
• Authors: Shuzi Niu, Jiafeng Guo, Yanyan Lan, Xueqi Cheng
• Keywords: Evaluation Measures, Information Technology, Relevance Judgments, Retrieval Models, SIGIR 2012