Reusable test collections through experimental design

Portable, reusable test collections are a vital part of research and development in information retrieval. Reusability is difficult to assess, however. The standard approach— simulating judgment collection when groups of systems are held out, then evaluating those held-out systems—only works when there is a large set of relevance judgments to draw on during the simulation. As test collections adapt to larger and larger corpora, it becomes less and less likely that there will be sufficient judgments for such simulation experiments. Thus we propose a methodology for information retrieval experimentation that collects evidence for or against the reusability of a test collection while judgments are being made. Using this methodology along with the appropriate statistical analyses, researchers will be able to estimate the reusability of their test collections while building them and implement“course corrections”if the collection does not seem to be achieving desired levels of reusa...

• Type: Conference
• Year: 2010
• Where: SIGIR
• Authors: Ben Carterette, Evangelos Kanoulas, Virgiliu Pavlu, Hui Fang
• Keywords: Information Management, Information Retrieval, Reusability, SIGIR 2010, Test Collection