Explaining User Performance in Information Retrieval: Challenges to IR Evaluation

The paper makes three points of significance for IR research: (1) The Cranfield paradigm of IR evaluation seems to lose power when one looks at human instead of system performance. (2) Searchers using IR systems in reallife use rather short queries, which individually often have poor performance. However, when used in sessions, they may be surprisingly effective. The searcher's strategies have not been sufficiently described and cannot therefore be properly understood, supported nor evaluated. (3) Searchers in real-life seek to optimize the entire information access process, not just result quality. Evaluation of output alone is insufficient to explain searcher behavior.

• Type: Journal
• Year: 2009
• Where: ICTIR
• Authors: Kalervo Järvelin
• Keywords: Entire Information Access, ICTIR 2009, Information Technology, IR Evaluation, IR Systems