Human versus machine in the topic distillation task

This paper reports on and discusses a set of user experiments using the TREC 2003 Web interactive track protocol. The focus is on comparing humans and machine algorithms in terms of performance in a topic distillation task. We also investigated the effect of the search results layout in supporting the users’ effort. We have demonstrated that machines can perform nearly as well as people on the topic distillation task. Given a system tailored to the task there is significant performance improvement and finally, given a presentation that supports the task, there is strong user satisfaction. Categories and Subject Descriptors H.3.3 [Information Storage and Retrieval]: Search Process, H.5.3 [Group and Organization Interfaces]: Evaluation, Webbased interaction, Organizational Design. General Terms Experimentation, Performance, Human Factors. Keywords Interactive information retrieval, topic distillation, search result organization, evaluation.

• Type: Conference
• Year: 2004
• Where: SIGIR
• Authors: Mingfang Wu, Gheorghe Muresan, Alistair McLean, Muh-Chyun (Morris) Tang, Ross Wilkinson, Yuelin Li, Hyuk-Jin Lee, Nicholas J. Belkin
• Keywords: SIGIR 2004, Topic Distillation, Topic Distillation Task, Trec 2003 Web