Improving Approximate Value Iteration Using Memories and Predictive State Representations

Planning in partially-observable dynamical systems is a challenging problem, and recent developments in point-based techniques such as Perseus significantly improve performance as compared to exact techniques. In this paper, we show how to apply these techniques to new models for nonMarkovian dynamical systems called Predictive State Representations (PSRs) and Memory-PSRs (mPSRs). PSRs and mPSRs are models of non-Markovian decision processes that differ from latent-variable models (e.g. HMMs, POMDPs) by representing state using only observable quantities. Further, mPSRs explicitly represent certain structural properties of the dynamical system that are also relevant to planning. We show how planning techniques can be adapted to leverage this structure to improve performance both in terms of execution time as well as quality of the resulting policy.

• Type: Conference
• Year: 2006
• Where: AAAI
• Authors: Michael R. James, Ton Wessling, Nikos A. Vlassis
• Keywords: AAAI 2006, Dynamical System, Intelligent Agents, NonMarkovian Dynamical Systems, Partially-observable Dynamical Systems