Add failsafe to reinforcement learning algorithm

Question

I'm working on a hexapod that uses A3C to learn how to walk. Ideally I would test it all in a simulator for some structure to the weights/policy but I don't have enough time for that. Obviously there are specific degrees of freedom that would hit each other at certain points, so how could I implement a failsafe that stopped certain movements without messing up the algorithm? If I were to just not allow a movement if I thought it would be dangerous after the algorithm but before the movement, would that disrupt it?

Answer 1

If I understand correctly, you are planning to use a real structure, instead of a simulation and you are woried about collisions of the legs of the robot during locomotion learning.

Yes, failsafes can and should be implemented. Workspace limits for each leg should be added to the control system.

A conservative and simple approach would be to add joint limits to each leg joints in a way that no collision in possible for any joint values. This would essentially mean that the volumes inside which the legs can move do not intersect eachother. Depending on the size (length) of your hexapod, this might mean serious restrictions or might not make a difference.

You can define joint limits dynamically, based on current values of the adjoining legs. I.e if the left forward legs first joint (I assume only first joints are mainly responsible for collisions) has a certain current (measured) value, the joint limits of the left middle legs first joint (again assuming these are mainly responsible for collisions) has a motion range expressed (restricted) in function of the first leg (and vice versa). If the front leg is in a frontal position the middle leg can move freely forward, however, if the first leg is in a rear position the middle leg can only move a limited amount forward. And vice-versa.

These dynamic limits will affect the RI algorithm, since there are motions which cannot be executed these have to be penalized by the loss function. I assume this will cause longer learning times. Furthermore it will also mean the your loss function will not only contain the reward for reaching the goal, but has to either model the joint limits exactly as your control system or get a feedback from your control system that the RI algorithm issued a command (action) which cannot be executed (woud lead to collision) and has to be penalized. I assume if the loss function takes care of the penalties, and the penalties are high enough, these limtis will not affect the state transitions of the robot, since these actions will never be further explored, since the penalty on them is so high.

There is however a different challenge, not just collisions. Your hexapod can fall down. Now this is a bit more harder to detect by the control system then the adding joint limits. It basically means that your control system will have to calculate the CoG of the robot and make sure that its projection remains inside the support polygon. This means that the control system wil have to simulate the robot motion, before carring out the action. If the action would lead to the robot falling down, again the control system will have to signal this to the loss function and the loss function has to penalize the actions.