I'm wondering if a single agent can train multiple robots to perform different tasks simultaneously. If possible, can you please recommend me some research papers and implementations that I can take a look at?
I was hoping we could create two gym environments like env1 = gym.make("robot1-task1") and env2 = gym.make("robot2-task2") then use a single agent to sample experience from both environments at the same time and generalize for both tasks.
Maybe.
Assuming what you are looking for here is some sort of actor-critic deep reinforcement learning setup, this will be tricky. I will try to list out the pitfalls I see, and maybe that will help you decide if this is worth it or not.
$\textit{Deep networks are smooth functions}$. This means that if two inputs $x_1$ and $x_2$ satisfy $|x_1 - x_2| \le \epsilon$ for some $\epsilon$, then $|f(x_1) - f(x_2)| \le \delta$ for some $\delta$, where $f(\cdot)$ is the output of your neural network. Let's frame this in terms of your actor network - the same argument would apply for a critic, but it's a bit more complicated yet does not add much to the discussion. So, you learn some policy $\pi(s)$, where $s$ is your current state. Assuming both robots have the same structure or at least the same number of actuators (we'll get into this later), the stated goal would be to produce two distinct (not necessarily different) sequence of activations. This is where things get hairy. Let's assume robot one is in state $s_1$ and robot two is in state $s_2$ and $s_1 \approx s_2$. It is reasonable to assume that robot one might maximize rewards by moving "left" and robot two by moving "right", thus one or both of the robots will clearly make an un-optimal move as the neural network activation will be very similar for the two states. Now if the two states are the same, we end up in an even worse case where, theoretically (chaos saves us here), both robots will have identical trajectories no matter what. How do we avoid this problem? You could perhaps add an additional state parameter to indicate which robot you are working with - in essence creating two distinct subspaces in your input space allowing your network to "decide" actions based on which task is being performed. This will likely require a large network though, and would probably still be hard to actually have the two robots perform distinct tasks. To frame the problem mathematically, you're essentially trying to learn something like a homotopy between two functions - which you are also trying to learn.
$\textit{Different tasks presumably have very different reward functions}$. Just to touch on issues with the critic network, the smooth nature of neural networks is even more detrimental here. It is reasonable to imagine the topology of different desired actuator activations are similar for different tasks. Take for example a torque activation of two similar robotic manipulators. The act of accounting for gravity for both manipulators is going to require similar torques given their similar structure. However, the rewards of different tasks are arbitrarily complicated and may or may not explicitly rely on manipulator structure (i.e. joint control vs. cartesian control). Thus learning the "homotopy" between the two reward functions will likely require even more neurons than the actor network did - with less guarantee of really being possible.
$\textit{The structure of your robots will need to be almost identical}$. This is basically a continuation of the previous two arguments. To learn separate tasks for the same manipulator via one set of networks $\textit{might}$ be possible. Learning these tasks with one distinct structure per task would likely require so many additional neurons that it would be hard to imagine a good reason for using this approach would exist.
My personal opinion on the matter would be to use a different set of networks for each task. If your wanting to switch between the two tasks for the same robot - develop a deterministic control sequence (i.e. hard-coded) way to do so. I guess you could even try to make a third RL structure to learn when to switch between the tasks if you wanted to. If you are concerned about training times and/or parallelism, I would say you probably won't lose much performance using two different network structures sequentially vs. one much, much larger network that still requires both environments to be run sequentially.
