Human performance and retrieval precision revisited

Several studies have found that the Cranfield approach to evaluation can report significant performance differences between retrieval systems for which little to no performance difference is found for humans completing tasks with these systems. We revisit the relationship between precision and performance by measuring human performance on tightly controlled search tasks and with user interfaces offering limited interaction. We find that human performance and retrieval precision are strongly related. We also find that users change their relevance judging behavior based on the precision of the results. This change in behavior coupled with the well-known lack of perfect inter-assessor agreement can reduce the measured performance gains predicted by increased precision. Categories and Subject Descriptors: H.3.3 [Information Search and Retrieval] General Terms: Experimentation, Human Factors, Performance

• Type: Conference
• Year: 2010
• Where: SIGIR
• Authors: Mark D. Smucker, Chandra Prakash Jethani
• Keywords: Human Performance, Information Management, Performance, Performance Difference, SIGIR 2010