Fractional Similarity: Cross-Lingual Feature Selection for Search

Abstract. Training data as well as supplementary data such as usagebased click behavior may abound in one search market (i.e., a particular region, domain, or language) and be much scarcer in another market. Transfer methods attempt to improve performance in these resourcescarce markets by leveraging data across markets. However, differences in feature distributions across markets can change the optimal model. We introduce a method called Fractional Similarity, which uses query-based variance within a market to obtain more reliable estimates of feature deviations across markets. An empirical analysis demonstrates that using this scoring method as a feature selection criterion in cross-lingual transfer improves relevance ranking in the foreign language and compares favorably to a baseline based on KL divergence.

• Type: Journal
• Year: 2011
• Where: ECIR
• Authors: Jagadeesh Jagarlamudi, Paul N. Bennett
• Keywords: ECIR 2011, Feature Selection, Information Technology, Optimal Model, Selection Criterion