Decision-making under uncertainty with multi-robot systems is essential for many applications. Real-world robotic sequential decision problems entail several challenges, including partial observation, dynamic environments, a large state space due to system complexity, and a large control space given by multiple agents with partial communication. We consider infinite-horizon discounted Markov decision problems under partial observation (POMDPs) with finite, discrete state and control spaces — where, for a team of 10 agents, the state space can reach 1037 states and the control space 107 controls.
Our work develops algorithms that use multi-step lookahead, truncated rollout with a known base policy, and terminal cost function approximation. These algorithms also serve for policy improvement in approximate policy iteration schemes, where successive policies are approximated using neural network classifiers. A novel feature of our approach is that it is well suited to distributed computation through a partitioned architecture trained with multiple neural networks. Our multiagent variants optimize the agents' controls one-at-a-time, which dramatically reduces the required computation while preserving the key cost improvement property of the standard methods: per-step computation grows linearly rather than exponentially with the number of agents.
We apply these methods to challenging problems including sequential repair, where robots collaboratively inspect and repair a pipeline under partial information about its state, and compare favorably with standard rollout and existing algorithms such as POMCP and DESPOT.