Least-Squares Temporal Difference Learning

Excerpted from: Boyan, Justin. Learning Evaluation Functions for Global Optimization. Ph.D. thesis, Carnegie Mellon University, August 1998. (Available as Technical Report CMU-CS-98-152.) TD ???? is a popular family of algorithms for approximate policy evaluation in large MDPs. TD ???? works by incrementally updating the value function after each observed transition. It has two major drawbacks: it makes inefficient use of data, and it requires the user to manually tune a stepsize schedule for good performance. For the case of linear value function approximations and ????? , the Least-Squares TD (LSTD) algorithm of Bradtke and Barto [5] eliminates all stepsize parameters and improves data efficiency. This paper extends Bradtke and Barto's work in three significant ways. First, it presents a simpler derivation of the LSTD algorithm. Second, it generalizes from ???? to arbitrary values of ? ; at the extreme of ?? , the resulting algorithm is shown to be a practical formulation of su...

• Type: Conference
• Year: 1999
• Where: ICML
• Authors: Justin A. Boyan
• Keywords: Approximate Policy Evaluation, BACKGROUND This Paper, ICML 1999, Linear Value Function, Machine Learning