1
$\begingroup$

Not sure if I am posting this question in the correct community, as it relates primarily to reinforcement learning. Apologies early on if this is not so.

In reinforcement learning many algorithms exist for 'solving' the cart-pole problem; that of balancing a mass on the edge of a stick, connected to a cart on a hinge, which has 1 DoF. There is TD learning, Q-learning and many other on and off-policy methods. There is also the more recent, model-based policy search method PILCO.

What I am really wondering, I suppose is more of a physics question: is there a need for active control? Why is it not possible to find the one point for the cart, which prevents the mass to move, even incrementally, left or right as it sits atop the pole? Why does it always 'fall'?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
5
$\begingroup$

Your question might be better suited to another forum, but is definitely important for robotics in general. You are basically talking about unstable equilibrium, where there does exist a point where the pole is balanced, but any small perturbation (random unforeseen input like wind or vibration) will cause it to diverge away from that balance.

This is one reason why active control is necessary. But even if the pole were a regular pendulum hanging down (with a stable equilibrium point), you still might want active control to stop it from swinging around too much. In the end, active control is what you need if you want to change the system dynamics -- whether that means going from unstable to stable or just improving the response for desired performance.

When you consider this in the context of reinforcement learning, you can think about active control as the input you still need for certain actions to account for effects that cannot be modelled. For example, driving a car involves learned actions that you simply just "do" without any real active control, like moving your eyes to look in the rear-view mirror. However, it also involves lots of active control to react to the changing environment (otherwise you could drive around blind-folded).

Take a look at combined feedback and feed-forward control, where the former is the active control and the latter representative of learned inputs.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks that was very informative, though suppose our entire goal is to just find the point at which the unstable equilibrium is reached. Can that point be reached and then kept there without need active control input? $\endgroup$
    – Astrid
    Nov 25, 2015 at 19:11
  • $\begingroup$ On paper, unstable equilibrium is still equilibrium but in reality it doesn't last very long. Imagine balancing a coin on its edge -- can be done but doesn't necessarily stay up long. Now imagine balancing a pencil on its sharp point -- almost impossible. For a cart with an inverted pendulum the equilibrium point is with the pendulum straight up (unless you are accelerating, or accounting for aerodynamic drag while moving at constant speed). But I would say in reality you will almost always need active control for an unstable system. $\endgroup$
    – Brian Lynch
    Nov 25, 2015 at 19:20
  • $\begingroup$ @astrid: how exactly do you think it could be "kept there"? In practice it is impossible to keep it there passively. There will always be disturbances. $\endgroup$
    – Paul
    Nov 25, 2015 at 19:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.