# Add failsafe to reinforcement learning algorithm

I'm working on a hexapod that uses A3C to learn how to walk. Ideally I would test it all in a simulator for some structure to the weights/policy but I don't have enough time for that. Obviously there are specific degrees of freedom that would hit each other at certain points, so how could I implement a failsafe that stopped certain movements without messing up the algorithm? If I were to just not allow a movement if I thought it would be dangerous after the algorithm but before the movement, would that disrupt it?

If I understand correctly, you are planning to use a real structure, instead of a simulation and you are woried about collisions of the legs of the robot during locomotion learning.

Yes, failsafes can and should be implemented. Workspace limits for each leg should be added to the control system.

A conservative and simple approach would be to add joint limits to each leg joints in a way that no collision in possible for any joint values. This would essentially mean that the volumes inside which the legs can move do not intersect eachother. Depending on the size (length) of your hexapod, this might mean serious restrictions or might not make a difference.

You can define joint limits dynamically, based on current values of the adjoining legs. I.e if the left forward legs first joint (I assume only first joints are mainly responsible for collisions) has a certain current (measured) value, the joint limits of the left middle legs first joint (again assuming these are mainly responsible for collisions) has a motion range expressed (restricted) in function of the first leg (and vice versa). If the front leg is in a frontal position the middle leg can move freely forward, however, if the first leg is in a rear position the middle leg can only move a limited amount forward. And vice-versa.

These dynamic limits will affect the RI algorithm, since there are motions which cannot be executed these have to be penalized by the loss function. I assume this will cause longer learning times. Furthermore it will also mean the your loss function will not only contain the reward for reaching the goal, but has to either model the joint limits exactly as your control system or get a feedback from your control system that the RI algorithm issued a command (action) which cannot be executed (woud lead to collision) and has to be penalized. I assume if the loss function takes care of the penalties, and the penalties are high enough, these limtis will not affect the state transitions of the robot, since these actions will never be further explored, since the penalty on them is so high.

There is however a different challenge, not just collisions. Your hexapod can fall down. Now this is a bit more harder to detect by the control system then the adding joint limits. It basically means that your control system will have to calculate the CoG of the robot and make sure that its projection remains inside the support polygon. This means that the control system wil have to simulate the robot motion, before carring out the action. If the action would lead to the robot falling down, again the control system will have to signal this to the loss function and the loss function has to penalize the actions.

• A dynamic failsafe is exactly what I'm looking for. I've never thought about the balance of the robot though, that seems like a serious issue. Would implementing a gyro help at all with that/would I have to create a formula to calculate the CoG? Commented May 8, 2017 at 15:10
• a gyro would tell you that the robot fell. However in that case you would need to interween (or implement a standup procedure). If you do not want to execute motions that would lead to the robot falling, I do not think there is a way around simulating the CoG motions before the step is taken. Maybe you could further add restriction about the number of legs in the air and their sequence to make sure that the support polygon is large enough, but those are a lot of artificial rules.
– 50k4
Commented May 8, 2017 at 15:12
• The one rule I was going to add was that only 3 legs could be in the air at a time (because that's the basis of hexapods). Falling down would also mess with the RL algo so I think simulations are a must. Would that be using Inverse Kinematics? Because I thought about doing something similar before but realized that I would just be writing the code for the robot to be able to manually walk, defeating the purpose of the RL. Commented May 8, 2017 at 15:22
• 3 legs in the air for not falling sound like a combined rule. There are more 3 leg combinations that can be lifted. However this will limit the full potential of the hexapod. If the body is pushed fully forward and one of the middle legs is pushed back there may be other possiblities for liftign legs
– 50k4
Commented May 8, 2017 at 15:25
• Depending of what is your purpose, Ik might help or not. IK doe snot tell you where to put your leg down, it just tells you which joint angles you need of you decided where to put the leg down. Why let RI learn the IK when it can be developed?
– 50k4
Commented May 8, 2017 at 15:26