Kalman Temporal Differences

This paper deals with value (and Q-) function approximation in deterministic Markovian decision processes (MDPs). A general statistical framework based on the Kalman filtering paradigm is introduced. Its principle is to adopt a parametric representation of the value function, to model the associated parameter vector as a random variable and to minimize the mean-squared error of the parameters conditioned on past observed transitions. From this general framework, which will be called Kalman Temporal Differences (KTD), and using an approximation scheme called the unscented transform, a family of algorithms is derived. Contrary to most of function approximation schemes, this framework inherently allows to derive uncertainty information over the value function, which can be notably useful for the exploration/exploitation dilemma.

• Type: Journal
• Year: 2010
• Where: JAIR
• Authors: Matthieu Geist, Olivier Pietquin
• Keywords: Approximation Schemes, Deterministic Markovian Decision, JAIR 2010, Value Function