Unifying Temporal and Structural Credit Assignment Problems

Single-agent reinforcement learners in time-extended domains and multi-agent systems share a common dilemma known as the credit assignment problem. Multi-agent systems have the structural credit assignment problem of determining the contributions of a particular agent to a common task. Instead, time-extended single-agent systems have the temporal credit assignment problem of determining the contribution of a particular action to the quality of the full sequence of actions. Traditionally these two problems are considered different and are handled in separate ways. In this article we show how these two forms of the credit assignment problem are equivalent. In this unified framework, a single-agent Markov decision process can be broken down into a single-time-step multi-agent process. Furthermore we show that Monte-carlo estimation or Q-learning (depending on whether the values of resulting actions in the episode are known at the time of learning) are equivalent to different agent uti...

• Type: Conference
• Year: 2004
• Where: ATAL
• Authors: Adrian K. Agogino, Kagan Tumer
• Keywords: ATAL 2004, Credit Assignment Problems, Multi-agent Systems, Structural Credit Assignment