Separating Precision and Mean in Dirichlet-Enhanced High-Order Markov Models

Abstract. Robustly estimating the state-transition probabilities of highorder Markov processes is an essential task in many applications such as natural language modeling or protein sequence modeling. We propose a novel estimation algorithm called Hierarchical Separated Dirichlet Smoothing (HSDS), where Dirichlet distributions are hierarchically assumed to be the prior distributions of the state-transition probabilities. The key idea in HSDS is to separate the parameters of a Dirichlet distribution into the precision and mean, so that the precision depends on the context while the mean is given by the lower-order distribution. HSDS is designed to outperform Kneser-Ney smoothing especially when the number of states is small, where Kneser-Ney smoothing is currently known as the state-of-the-art technique for N-gram natural language models. Our experiments in protein sequence modeling showed the superiority of HSDS both in perplexity evaluation and classification tasks.

• Type: Conference
• Year: 2007
• Where: ECML
• Authors: Rikiya Takahashi
• Keywords: Dirichlet Distribution, ECML 2007, Machine Learning, Protein Sequence Modeling, Separated Dirichlet Smoothing