0
$\begingroup$

I am trying to understand optimal control theory which forms the base for reinforcement learning techniques in AI. Whenever I open a lecture or a book or any online notes, everything starts with an ODE and then derivation goes the payoff function which is straight forward.

I am trying hard to comprehend why an ODE models any system? Many say it easy to begin with but why this model?

$dx/dt = f(x(t))$

I could not find the reason and decided to ask for help.

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ I don't understand your question. You need ODEs to mathematically describe how the system behaves as time goes. What else need to be said?! May be if you narrow your question a bit, ppl may help. $\endgroup$
    – CroCo
    Feb 16, 2017 at 14:30
  • $\begingroup$ ODE stands for Ordinary differential equation which describes analog systems like curves. For controlling a robot that is not powerfull enough. So hybrid systems are extended by with finite state machines. On the one hand, you have a normal state-machine which iterates through the walk cycles and on the other hand inside every state a solver calculates Ordinary differential equations. (1) $\endgroup$
    – Manuel Rodriguez
    Feb 23, 2017 at 21:08
  • $\begingroup$ Reinforcement Learning definitely isn't based upon continuous time optimal control theory. $\endgroup$
    – FourierFlux
    Jul 25, 2020 at 23:44

2 Answers 2

Reset to default
1
$\begingroup$

The reason why ODE's are used is simply: physics. It would be great if any system could be modelled by a simple linear function like $x(t)=at$, but nature is not so simple, or linear. Even when you neglect nature, dynamical systems, like $\dot{x}(t)=f(x(t))$ still pop up everywhere, like CroCo said, it is the basis of the mathematical modelling of many systems.

I would suggest looking into differential equations first before starting with reinforcement learning.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I looked at one f the differential equation class on edx. Cleared it up. Thanks. $\endgroup$
    – bicepjai
    Feb 16, 2017 at 19:29
  • 1
    $\begingroup$ @bicepjai: What exactly was cleared up by the edx class? Please share the details of what you learned from the class that clarified your question. $\endgroup$
    – Paul
    Feb 16, 2017 at 21:35
  • $\begingroup$ Parameters on modeling depends on the system we are attempting to model which is f(x) here. Rate of change in a system is a natural way of modeling which gives us the necessary system state as function of time, convinced myself after sifting through examples. I ended up signing up for the mitx diff eqn course. $\endgroup$
    – bicepjai
    Feb 16, 2017 at 21:44
1
$\begingroup$

Appending to @WalterJ's answer.

Linear and nonlinear systems which form the basis for subjects like optimal control theory have rigorous math fundamentals which allow you to analyze ODEs without actually solving them and mathematically prove whether is system is stable, how fast your convergence will be or define a safe operating region. This makes representing systems as ODEs very useful in control systems. ODE representation also makes it clear how your states (eg: position, velocity, acceleration for cars) interact with each other (is there any coupling between the states?).

Coming back to optimal control, the cost functional is expressed as functions of your states and/or control inputs. Say you want to minimize distance, your cost functional will be the integral over the velocity for all time. As you expressed your system as an ODE, you already have access to this expression.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.