How to use a POMDP-based planner on top of a probabilistic filter

Question

POMDPs extend MDPs by conceiling state and adding an observation model. A POMDP controller processes either

• action/observation histories or
• a bayesian belief state, computed from the observations (belief-MDP transformation)

In a complex, real-world system like a robot, one usually preprocesses sensory readings using filters (Kalmann, HMM, whatever). The result of which is a belief-state.

I am looking for publications that discuss the problem of fitting a (probably more abstract) POMDP model on top of an existing filter-bank.

1. Do you have to stick to the belief-MDP, and hand over the filtered belief-state to the controller?
2. Is there any way of using history-based POMDP controllers, like MCTS?
3. How do you construct/find the abstract observations you need to formulate the POMDP model?

Answer 1

I have used POMDP like models on top of a localization algorithm (Adaptive Monte Carlo Localization, from ROS), and a person detector [1][2] to find and follow a person with a humanoid robot. These two algorithms generate the input (observation) for the POMDP model in [1] and [2]. Also in [3] they used a POMDP model with similar input.

As next step we used POMCPs (Partially Observable Monte Carlo Processes [4]) which use Monte Carlo simulations to create a policy because they are able to handle much larger state spaces. Having continuous states and observations causes problems when trying to find a policy, since there will be an infinite number of observations possible. We used Monte Carlo processes with continuous states [4], but the observations and actions were discretized to prevent the policy tree from growing too wide. There is however work on the use of continuous observations in POMDP, such as presented in [5].

Other work with POMDP like models on a top level are: [6-8].

1. A. Goldhoorn, A. Sanfeliu and R. Alquézar Mancho. Analysis of methods for playing human robot hide-and-seek in a simple real world urban environment, 1st Iberian Robotics Conference, 2013, Madrid, in ROBOT2013: First Iberian Robotics Conference, Vol 252-3 of Advances in Intelligent Systems and Computing, pp. 505-520, 2014, Springer. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7041445
2. A. Goldhoorn, A. Garrell Zulueta, R. Alquézar Mancho and A. Sanfeliu. Continuous real time POMCP to find-and-follow people by a humanoid service robot, 2014 IEEE-RAS International Conference on Humanoid Robots, 2014, Madrid, Spain, pp. 741-747, IEEE Press. http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7041445
3. Luis Merino, Joaqu\'in Ballesteros, Noé Pérez-Higueras, Rafael Ramón-Vigo, Javier Pérez-Lara, and Fernando Caballero. Robust Person Guidance by Using Online POMDPs. In Manuel A. Armada, Alberto Sanfeliu, and Manuel Ferre, editors, ROBOT2013: First Iberian Robotics Conference, Advances in Intelligent Systems and Computing, pp. 289–303, Springer International Publishing, 2014. http://link.springer.com/chapter/10.1007%2F978-3-319-03653-3_22
4. D. Silver and J. Veness, “Monte-Carlo planning in large POMDPs,” Proceedings of 24th Advances in Neural Information Processing Systems (NIPS), pp. 1–9, 2010.
5. J.M. Porta, N. Vlassis, M.T. Spaan and P. Poupart. Point-based value iteration for continuous POMDPs. Journal of Machine Learning Research, 7: 2329-2367, 2006.
6. Pineau, J., Gordon, G., & Thrun, S. (2003). Point-based value iteration: An anytime algorithm for POMDPs. In IJCAI International Joint Conference on Artificial Intelligence (Vol. 18, pp. 1025–1030). Citeseer. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.68.1777&rep=rep1&type=pdf
7. Ong, S., Png, S., & Hsu, D. (2009). POMDPs for robotic tasks with mixed observability. Proc. Robotics: Science and Systems. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.149.3849&rep=rep1&type=pdf
8. Schesvold, D., Tang, J., Ahmed, B. M., Altenburg, K., & Nygard, K. E. (2003). POMDP planning for high level UAV decisions: Search vs. strike. In In Proceedings of the 16th International Conference on Computer Applications in Industry and Engineering (pp. 3–6). Citeseer. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.125.9134