what are the main steps to build a swarm robots system and train it to achieve foraging task using deep Q network

Question

I studied reinforcement learning deeper and prepared myself to use Webots, and when I decided to build a swarm robots system and drive it by deep_Q_networks I feel too confused how can I begin and is there any tutorial that can help that includes DQN,swarm,and Webots

Answer 1

Let's assume this foraging task is a continuous motion planning problem, i.e. you have dynamics accounted for (does not necessarily have to be the case, but should help convergence if it is) and your available actions are some movements relative to each robots current position. I would first advise ditching DQN. Learning how DQN's work is an important first step in reinforcement learning, but a more modern solution like SAC does not add much computational complexity while providing vast improvements (also it uses a continuous action space which tends to be a win in robotics). So, the next question is how do I use SAC? Pretty easy here, go to a repo like this https://github.com/denisyarats/pytorch_sac and clone it. You do not need to be an expert at all on the inner workings of these RL algorithms right away. Give yourself some time using them and build up an intuitive understanding of what is going on. It takes time to learn them as PyTorch and Tensorflow are complex libraries that do a lot under the hood.

Now, you have your RL code ready to go, how do you actually apply it to your problem? This is where the actual engineering/math comes in. First, you'll probably want to set up a simulation environment with all of your robots. You'll want a function that returns the state of all of your robots in a long vector, $\mathbf{s} \in \mathbb{R}^{e\cdot |s_i|\times 1}$ where $|s_i|$ represents the number of states of an individual robot and $e$ represents how many robots you have. You will also need an adequate reward function. This is the most important part of any RL algorithm. Your reward here would obviously be based on foraging performance, but I would imagine the swarm aspect of the problem means you are looking to maximize coverage over some space? Making this assumption, you could have a reward that is simply $r(\mathbf{s}) = f(\mathbf{s}) + A(\mathbf{s})$, where $f(\mathbf{s})$ denotes the foraging performance of the swarm and $A(\mathbf{s})$ represents the area they are covering. Keep in mind you may also want to include some statistics about foraging performance in your state then.

Now, you have a reward function and a state function - the final step is to make a step function. This will take you from your current state to another state given some action sampled from your network. So, we will need an action vector $\mathbf{a} \in \mathbb{R}^{e \cdot |a_i| \times 1}$, where $|a_i|$ represents the number of available actions per robot. So, the step function will look something like $(\mathbf{s}_{t+1}, r_{t}, d) = step(\mathbf{s}_t, \mathbf{a}_t)$, where $d \in \{0, 1\}$ indicates the termination status of your current run (not necessarily the whole simulation, just one of many runs needed to collect training data) and $t$ represents the current time step.

For convenience sake, most RL implementations are set up to be interfaced with OpenAI Gym environments. Thus, if you make the above functions, you can simply "inject" your own environment into many pre-existing RL implementation and get going in a relatively smooth manner. The Gym interface typically only requires a $step$ and $reset$ (just resets environment state to some default) function as well as perhaps some other values that should be apparent by any error messages ;) In my personal experience, the actual coding of the environment is much more tedious (albeit rewarding) than interfacing the environment with a prefab RL implementation.

Various sources to help in your journey:

• Reward function crafting:
  • https://stats.stackexchange.com/questions/189067/how-to-make-a-reward-function-in-reinforcement-learning
  • https://www.mathworks.com/videos/reinforcement-learning-part-2-understanding-the-environment-and-rewards-1551976590603.html
  • https://people.eecs.berkeley.edu/~pabbeel/cs287-fa09/readings/NgHaradaRussell-shaping-ICML1999.pdf (this one is a little dense)
• RL Code and Theory:
  • https://github.com/sweetice/Deep-reinforcement-learning-with-pytorch
  • https://github.com/aikorea/awesome-rl
  • https://github.com/nicklashansen/tdmpc (relatively "cutting-edge" here)