Adaptive term frequency normalization for BM25

A key component of BM25 contributing to its success is its sub-linear term frequency (TF) normalization formula. The scale and shape of this TF normalization component is controlled by a parameter k1, which is generally set to a term-independent constant. We hypothesize and show empirically that in order to optimize retrieval performance, this parameter should be set in a term-specific way. Following this intuition, we propose an information gain measure to directly estimate the contributions of repeated term occurrences, which is then exploited to fit the BM25 function to

• Type: Journal
• Year: 2011
• Where: CIKM
• Authors: Yuanhua Lv, ChengXiang Zhai
• Keywords: CIKM 2011, Information Gain, Information Technology, Retrieval Performance, Term Occurrences