Intuition-supporting visualization of user's performance based on explicit negative higher-order relevance

Modeling the beyond-topical aspects of relevance are currently gaining popularity in IR evaluation. For example, the discounted cumulated gain (DCG) measure implicitly models some aspects of higher-order relevance via diminishing the value of relevant documents seen later during retrieval (e.g., due to information cumulated, redundancy, and effort). In this paper, we focus on the concept of negative higher-order relevance (NHOR) made explicit via negative gain values in IR evaluation. We extend the computation of DCG to allow negative gain values, perform an experiment in a laboratory setting, and demonstrate the characteristics of NHOR in evaluation. The approach leads to intuitively reasonable performance curves emphasizing, from the user's point of view, the progression of retrieval towards success or failure. We discuss normalization issues when both positive and negative gain values are allowed and conclude by discussing the usage of NHOR to characterize test collections. Ca...

• Type: Journal
• Year: 2008
• Where: SIGIR
• Authors: Heikki Keskustalo, Kalervo Järvelin, Ari Pirkola, Jaana Kekäläinen
• Keywords: Beyond-topical Aspects, Information Technology, IR Evaluation, Negative Gain Values, SIGIR 2008