If I Had a Million Queries

As document collections grow larger, the information needs and relevance judgments in a test collection must be well-chosen within a limited budget to give the most reliable and robust evaluation results. In this work we analyze a sample of queries categorized by length and corpus-appropriateness to determine the right proportion needed to distinguish between systems. We also analyze the appropriate division of labor between developing topics and making relevance judgments, and show that only a small, biased sample of queries with sparse judgments is needed to produce the same results as a much larger sample of queries.

• Type: Conference
• Year: 2009
• Where: ECIR
• Authors: Ben Carterette, Virgiliu Pavlu, Evangelos Kanoulas, Javed A. Aslam, James Allan
• Keywords: Computer Science, ECIR 2009, Larger Sample, Relevance Judgments, Sparse Judgments