I am trying to implement a RL algorithm for an adaptive PID in a robot system. My doubt consists in the creation of the possible states in the problem.
I mean, I understand quite well the problem when the possible states are door numbers, but I don't know know what to do with PID. Do I have to create a finite number of possible PID values in which the algorithm learns?
I read a bit more and realized that in RL states and rewards accept a wide variety of interpretations and this is the real complexity nowadays of this learning problem.
In case of PID values, problem can be formulated as the following:
imagine a Kp value, it represents a state. Next state could be increase or decrease 0.1. Same with the next state, and so on.
Moreover, in some research problems, when talking not about controllers, but about external efforts, they simplify the problem to three possible states: fixed positive value, fixed negative value or 0 value. It is complicated to apply this idea to controllers.
I am almost sure that my particular problem could be more easily formulated if I used as states another kind of variables, such as "advancing forward, backwards or not moving". RL allows it.
Many reainforcement learning methods require descrete actions. As you indentified, increasing and decreasing the values is one option. If it is an adaptive PID, then it might take some time to incerase the parameters if you only have an increase by the factor of 0.1. I would recommend more then one increasing factor as possible action. Increase by 0.1, 1, 10, 20, 50, 100 etc. Also, increasing with a certain amount might also help you.
There is however a different option. The Deep Deterministic Policy Gradient approach can deal with continous states, so using this method, there is no need for discretization. You can find more information here and here
