Query polyrepresentation for ranking retrieval systems without relevance judgments

Ranking information retrieval (IR) systems with respect to their effectiveness is a crucial operation during IR evaluation, as well as during data fusion. This paper offers a novel method of approaching the system ranking problem, based on the widely studied idea of polyrepresentation. The principle of polyrepresentation suggests that a single information need can be represented by many query articulations–what we call query aspects. By skimming the top k (where k is small) documents retrieved by a single system for multiple query aspects, we collect a set of documents that are likely to be relevant to a given test topic. Labeling these skimmed documents as putatively relevant lets us build pseudo-relevance judgments without undue human intervention. We report experiments where using these pseudo-relevance judgments delivers a rank ordering of IR systems that correlates highly with rankings based on human relevance judgments. 1 

• Type: Journal
• Year: 2010
• Where: JASIS
• Authors: Miles Efron, Megan A. Winget
• Keywords: JASIS 2010, Pseudo-relevance Judgments, Query Aspects, Rankings