# Dynamic programming algorithm aka Bellman equation in Robotics?

The dynamic programming algorithm refers to the Bellman equation. An open-loop control decides movement at the initial point while a closed-loop control decides control during the movement. Now most robotic application looks like closed-loop control: in every point, it checks how it is doing with respect to some reward function, this is my thinking. Now most participants in threads such as How mature is real-time programming in robotics? do not differentiate their scope, perhaps they haven't thought about it. Anyway, I am interested to know:

How is dynamic programming used in robotics? Is there any research about DP usage in robotics?

• How is "dynamic programming" related to "real-time programming"? – Shahbaz Apr 3 '13 at 8:49
• @Shahbaz "Dynamic programming" is a term invented by Bellman, a specific mathematical thing used in academia. The "real-time programming" may be or may not be related to it, I don't know. It looks like a term used by some people not understanding some applied mathematics. Can you explain me what you mean by real-time programming? It looks like closed-loop optimization problem. – hhh Apr 3 '13 at 12:30
• Real-time computing is not a term used by people who don't understand applied mathematics. Real-time computing/programming is a part of computer science related to implementing and scheduling tasks that have a deadline. In particular, hard real-time tasks should not miss their deadlines under any circumstances. Real-time is often mistaken for "fast", which is an orthogonal concept. Closed-loop control is a very simple example of a real-time task. Dynamic program is a completely unrelated subject. – Shahbaz Apr 3 '13 at 13:18

I won't answer directly how Bellman equation is used in robotics but DP is used in many areas, see the below. Even though we have a dynamic system with velocity/acceleration, we can still use linear-approximation like linear-programming algorithms here. I provide below some material and lectures about Dynamic programming and robotics. This is pretty interdisciplinary area so hard to find concise description.

Areas

• path-planning
• vision such as stereo-vision
• CSP and VCSP problems (see the cognitive robotics below)

The term "dynamic programming" is a special term referring to special class of mathematical problems. The term "dynamic" system in physics refers to problems in continuum mechanics where objects are not in rest or in constant motion but they can accelerate. Then again Statics considers problems in stable equilibrium.

The term "dynamic programming" is programming for historical reasons but the basic idea is to break the large problem into small problems. It has analogues to fields such as Economics (for example sub-game-perfect equilibrium in game-theory).

Material

Path-planning

Vision, Stereo vision and Dynamic programming

Lectures

• Underactuated robotics in MIT here -- this lecture here about DP