Asymptotics of the Entropy Rate for a Hidden Markov Process

We calculate the Shannon entropy rate of a binary Hidden Markov Process (HMP), of given transition rate and noise (emission), as a series expansion in . The first two orders are calculated exactly. We then evaluate, for finite histories, simple upper-bounds of Cover and Thomas. Surprisingly, we find that for a fixed order k and history of n steps, the bounds become independent of n for large enough n. This observation is the basis of a conjecture, that the upper-bound obtained for n (k + 3)/2 gives the exact entropy rate for any desired order k of .

• Type: Conference
• Year: 2005
• Where: DCC
• Authors: Or Zuk, Ido Kanter, Eytan Domany
• Keywords: Binary Hidden Markov, Computer Graphics, DCC 2005, Exact Entropy Rate, Shannon Entropy Rate