Sciweavers

SIGIR
2003
ACM

On the effectiveness of evaluating retrieval systems in the absence of relevance judgments

14 years 4 months ago
On the effectiveness of evaluating retrieval systems in the absence of relevance judgments
Soboroff, Nicholas and Cahan recently proposed a method for evaluating the performance of retrieval systems without relevance judgments. They demonstrated that the system evaluations produced by their methodology are correlated with actual evaluations using relevance judgments in the TREC competition. In this work, we propose an explanation for this phenomenon. We devise a simple measure for quantifying the similarity of retrieval systems by assessing the similarity of their retrieved results. Then, given a collection of retrieval systems and their retrieved results, we use this measure to assess the average similarity of a system to the other systems in the collection. We demonstrate that evaluating retrieval systems according to average similarity yields results quite similar to the methodology proposed by Soboroff et al., and we further demonstrate that these two techniques are in fact highly correlated. Thus, the techniques are effectively evaluating and ranking retrieval syste...
Javed A. Aslam, Robert Savell
Added 05 Jul 2010
Updated 05 Jul 2010
Type Conference
Year 2003
Where SIGIR
Authors Javed A. Aslam, Robert Savell
Comments (0)