Learning Concept Importance Using a Weighted Dependence Model

Modeling query concepts through term dependencies has been shown to have a significant positive effect on retrieval performance, especially for tasks such as web search, where relevance at high ranks is particularly critical. Most previous work, however, treats all concepts as equally important, an assumption that often does not hold, especially for longer, more complex queries. In this paper, we show that one of the most effective existing term dependence models can be naturally extended by assigning weights to concepts. We demonstrate that the weighted dependence model can be trained using existing learning-to-rank techniques, even with a relatively small number of training queries. Our study compares the effectiveness of both endogenous (collectionbased) and exogenous (based on external sources) features for determining concept importance. To test the weighted dependence model, we perform experiments on both publicly available TREC corpora and a proprietary web corpus. Our experime...

• Type: Conference
• Year: 2010
• Where: WSDM
• Authors: Michael Bendersky, Donald Metzler, W. Bruce Croft
• Keywords: Data Mining, Keywords Weighted Dependence, Term Dependence Model, Weighted Dependence Model, WSDM 2010