Principled Methods for Advising Reinforcement Learning Agents

An important issue in reinforcement learning is how to incorporate expert knowledge in a principled manner, especially as we scale up to real-world tasks. In this paper, we present a method for incorporating arbitrary advice into the reward structure of a reinforcement learning agent without altering the optimal policy. This method extends the potentialbased shaping method proposed by Ng et al. (1999) to the case of shaping functions based on both states and actions. This allows for much more specific information to guide the agent ? which action to choose ? without requiring the agent to discover this from the rewards on states alone. We develop two qualitatively different methods for converting a potential function into advice for the agent. We also provide theoretical and experimental justifications for choosing between these advice-giving algorithms based on the properties of the potential function.

• Type: Conference
• Year: 2003
• Where: ICML
• Authors: Eric Wiewiora, Garrison W. Cottrell, Charles Elkan
• Keywords: ICML 2003, Machine Learning, Potential Function, Potentialbased Shaping Method, Reinforcement Learning Agent