I am training a reinforcement learning network in simulation for a robot which at the current stage learns Euler Angles to rotate the end-effector based on the actual state. The performance is overall not that satisfying. My network architecture is rather small and contains only two hidden layers. So, I would like to know what type of rotations can best be learned by neural networks? Euler Angles, Rotation matrices or Quaternions?
If someone could recommend a publication or paper for this topic, I would be very interested. Thank you for your help!