Counter Example for Q-Bucket-Brigade Under Prediction Problem

Aiming to clarify the convergence or divergence conditions for Learning Classifier System (LCS), this paper explores: (1) an extreme condition where the reinforcement process of LCS diverges; and (2) methods to avoid such divergence. Based on our previous work that showed equivalence between LCS’s reinforcement process and Reinforcement Learning (RL) with Function approximation (FA) method, we present a counter-example for LCS with Q-bucketbrigade based on the 11-state star problem, a counterexample originally proposed to show the divergence of Qlearning with linear FA. Furthermore, the empirical results applying the counter-example to LCS verified the results predicted from the theory: (1) LCS with Q-bucket-brigade diverged under the prediction problem, where the action selection policy was fixed; and (2) such divergence was avoided by using implicit-bucket-brigade or applying residual gradient algorithm to Q-bucket-brigade. Categories and Subject Descriptors I.2.6 [Artificial ...

• Type: Conference
• Year: 2005
• Where: IWLCS
• Authors: Atsushi Wada, Keiki Takadama, Katsunori Shimohara
• Keywords: IWLCS 2005, LCS’s Reinforcement Process, Machine Learning, Reinforcement Learning, Reinforcement Process