Finite-sample Analysis of Bellman Residual Minimization

We consider the Bellman residual minimization approach for solving discounted Markov decision problems, where we assume that a generative model of the dynamics and rewards is available. At each policy iteration step, an approximation of the value function for the current policy is obtained by minimizing an empirical Bellman residual defined on a set of n states drawn i.i.d. from a distribution

• Type: Journal
• Year: 2010
• Where: JMLR
• Authors: Odalric-Ambrym Maillard, Rémi Munos, Alessandro Lazaric, Mohammad Ghavamzadeh
• Keywords: Bellman Residual Minimization, Empirical Bellman, JMLR 2010, Value Functions