Learning Rates for Q-Learning

In this paper we derive convergence rates for Q-learning. We show an interesting relationship between the convergence rate and the learning rate used in Q-learning. For a polynomial learning rate, one which is 1/tω at time t where ω ∈ (1/2,1), we show that the convergence rate is polynomial in 1/(1 − γ), where γ is the discount factor. In contrast we show that for a linear learning rate, one which is 1/t at time t, the convergence rate has an exponential dependence on 1/(1−γ). In addition we show a simple example that proves this exponential behavior is inherent for linear learning rates.

• Type: Conference
• Year: 2001
• Where: COLT
• Authors: Eyal Even-Dar, Yishay Mansour
• Keywords: COLT 2001, Convergence Rate, Learning Rates, Linear Learning Rates, Machine Learning