Batch Reinforcement Learning with State Importance

Abstract. We investigate the problem of using function approximation in reinforcement learning where the agent’s policy is represented as a classifier mapping states to actions. High classification accuracy is usually deemed to correlate with high policy quality. But this is not necessarily the case as increasing classification accuracy can actually decrease the policy’s quality. This phenomenon takes place when the learning process begins to focus on classifying less “important” states. In this paper, we introduce a measure of state’s decision-making importance that can be used to improve policy learning. As a result, the focused learning process is shown to converge faster to better policies1 . 1 Problem Formulation and Related Work Reinforcement learning (RL) [11] provide a general framework for many sequential decision-making problems and has succeeded in a number of important applications. Let S be the state space, A the action set, and D the start-state distribution....

• Type: Conference
• Year: 2004
• Where: ECML
• Authors: Lihong Li, Vadim Bulitko, Russell Greiner
• Keywords: ECML 2004, Learning Process, Policy, Reinforcement Learning