Evaluation in Context

Abstract. All search happens in a particular context—such as the particular collection of a digital library, its associated search tasks, and its associated users. Information retrieval researchers usually agree on the importance of context, but they rarely address the issue. In particular, evaluation in the Cranfield tradition abstracting away from individual differences between users. This paper investigates if we can bring some of this context into the Cranfield paradigm. Our approach is the following: we will attempt to record the “context” of the humans already in the loop—the topic authors/assessors—by designing targeted questionnaires. The questionnaire data becomes part of the evaluation test-suite as valuable data on the context of the search requests. We have experimented with this questionnaire approach during the evaluation campaign of the INitiative for the Evaluation of XML Retrieval (INEX). The results of this case study demonstrate the viability of the quest...

• Type: Conference
• Year: 2009
• Where: ERCIMDL
• Authors: Jaap Kamps, Mounia Lalmas, Birger Larsen
• Keywords: Associated Search Tasks, Education, ERCIMDL 2009, Evaluation, Questionnaire Approach