Efficient Reinforcement Learning Using Recursive Least-Squares Methods

The recursive least-squares (RLS) algorithm is one of the most well-known algorithms used in adaptive filtering, system identification and adaptive control. Its popularity is mainly due to its fast convergence speed, which is considered to be optimal in practice. In this paper, RLS methods are used to solve reinforcement learning problems, where two new reinforcement learning algorithms using linear value function approximators are proposed and analyzed. The two algorithms are called RLS-TD( ) and Fast-AHC (Fast Adaptive Heuristic Critic), respectively. RLS-TD( ) can be viewed as the extension of RLS-TD(0) from =0 to general 0 1, so it is a multi-step temporal-difference (TD) learning algorithm using RLS methods. The convergence with probability one and the limit of convergence of RLS-TD( ) are proved for ergodic Markov chains. Compared to the existing LS-TD( ) algorithm, RLS-TD( ) has advantages in computation and is more suitable for online learning. The effectiveness of RLS-...

• Type: Journal
• Year: 2002
• Where: JAIR
• Authors: Xin Xu, Hangen He, Dewen Hu
• Keywords: Adaptive Heuristic Critic, Algorithm, JAIR 2002, RLS Methods