A Fast Analytical Algorithm for Solving Markov Decision Processes with Real-Valued Resources

Agents often have to construct plans that obey deadlines or, more generally, resource limits for real-valued resources whose consumption can only be characterized by probability distributions, such as execution time or battery power. These planning problems can be modeled with continuous state Markov decision processes (MDPs) but existing solution methods are either inefficient or provide no guarantee on the quality of the resulting policy. We therefore present CPH, a novel solution method that solves the planning problems by first approximating with any desired accuracy the probability distributions over the resource consumptions with phasetype distributions, which use exponential distributions as building blocks. It then uses value iteration to solve the resulting MDPs by exploiting properties of exponential distributions to calculate the necessary convolutions accurately and efficiently while providing strong guarantees on the quality of the resulting policy. Our experimental fe...

• Type: Conference
• Year: 2007
• Where: IJCAI
• Authors: Janusz Marecki, Sven Koenig, Milind Tambe
• Keywords: Artificial Intelligence, Exponential Distributions, IJCAI 2007, Probability Distributions, Solution Method