Sciweavers

SIGIR
1998
ACM

How Reliable Are the Results of Large-Scale Information Retrieval Experiments?

14 years 3 months ago
How Reliable Are the Results of Large-Scale Information Retrieval Experiments?
Two stages in measurement of techniques for information retrieval are gathering of documents for relevance assessment and use of the assessments to numerically evaluate effectiveness. We consider both of these stages in the context of the TREC experiments, to determine whether they lead to measurements that are trustworthy and fair. Our detailed empirical investigation of the TREC results shows that the measured relative performance of systems appears to be reliable, but that recall is overestimated: it is likely that many relevant documents have not been found. We propose a new pooling strategy that can significantly increase the number of relevant documents found for given effort, without compromising fairness.
Justin Zobel
Added 05 Aug 2010
Updated 05 Aug 2010
Type Conference
Year 1998
Where SIGIR
Authors Justin Zobel
Comments (0)