Evaluating aggregated search pages

Aggregating search results from a variety of heterogeneous sources or verticals such as news, image and video into a single interface is a popular paradigm in web search. Although various approaches exist for selecting relevant verticals or optimising the aggregated search result page, evaluating the quality of an aggregated page is an open question. This paper proposes a general framework for evaluating the quality of aggregated search pages. We evaluate our approach by collecting annotated user preferences over a set of aggregated search pages for 56 topics and 12 verticals. We empirically demonstrate the fidelity of metrics instantiated from our proposed framework by showing that they strongly agree with the annotated user preferences of pairs of simulated aggregated pages. Furthermore, we show that our metrics agree with the majority user preference more often than the current diversity-based information retrieval metrics. Finally, we demonstrate the flexibility of our framework...

• Type: Journal
• Year: 2012
• Where: SIGIR
• Authors: Ke Zhou, Ronan Cummins, Mounia Lalmas, Joemon M. Jose
• Keywords: Heterogeneous Sources, Information Technology, Preference Data, Search Evaluation, SIGIR 2012