Hierarchical pitman-yor language model for information retrieval

In this paper, we propose a new application of Bayesian language model based on Pitman-Yor process for information retrieval. This model is a generalization of the Dirichlet distribution. The Pitman-Yor process creates a power-law distribution which is one of the statistical properties of word frequency in natural language. Our experiments on Robust04 indicate that this model improves the document retrieval performance compared to the commonly used Dirichlet prior and absolute discounting smoothing techniques. Categories and Subject Descriptors: H.3.3 [Information Storage and Retrieval]:Information Search and Retrieval General Terms: Theory, Algorithm, Experimentation

• Type: Conference
• Year: 2010
• Where: SIGIR
• Authors: Saeedeh Momtazi, Dietrich Klakow
• Keywords: Bayesian Language Model, Dirichlet Distribution, Information Management, Pitman-Yor Process, SIGIR 2010