Evaluating web search using task completion time

We consider experiments to measure the quality of a web search algorithm based on how much total time users take to complete assigned search tasks using that algorithm. We first analyze our data to verify that there is in fact a negative relationship between a user’s total search time and a user’s satisfaction for the types of tasks under consideration. Secondly, we fit models with the user’s total search time as the response to compare two different search algorithms. The two search algorithms we chose for comparison are close in quality, but still differ enough that other evaluation methods have had some degree of success in separating them. We confirm that our methodology is in fact sensitive enough to detect that one of these two algorithms has a statistically significant speed advantage over the other. Finally, we propose an alternative experiential design which we demonstrate to be a substantial improvement over our current design in terms of variance reduction and e...

• Type: Conference
• Year: 2009
• Where: SIGIR
• Authors: Ya Xu, David Mease
• Keywords: Information Retrieval, Search Algorithms, SIGIR 2009, Total Search Time, User’s Total Search