Robotics Stack Exchange is a question and answer site for professional robotic engineers, hobbyists, researchers and students. It only takes a minute to sign up.
Sign up to join this community
I'm relatively new to the area. I run into several time the term ``rollout'' in training neural networks. I have been searching for a while but still not sure what it means.
For example, the number of rollout for running the hopper environment. I'm not sure what it means.
The standard use of “rollout” (also called a “playout”) is in regard to an execution of a policy from the current state when there is some uncertainty about the next state or outcome - it is one simulation from your current state. The purpose is for an agent to evaluate many possible next actions in order to find an action that will maximize value (long-term expected reward).
Uncertainty in the next state can arise from different sources depending on your domain. In games the uncertainty is typically from your opponent (you are not certain what move they will make next) or a chance element (e.g. a dice roll). In robotics, you may be modeling uncertainty in your environment (e.g. your perception system gives inaccurate pose estimates, so you are not sure an object is where you think it is) or your robot (e.g. noisy sensors result in unreliable transition dynamics).
I think the term comes from Tesauro and Galperin NIPS 1997 in which they consider Monte Carlo simulations of Backgammon where a playout considers a sequence of dice rolls:
In backgammon parlance, the expected value of a position is known as the "equity" of the position, and estimating the equity by Monte-Carlo sampling is known as performing a "rollout." This involves playing the position out to completion many times with different random dice sequences, using a fixed policy P to make move decisions for both sides.
As you could imagine, this can be quite computationally expensive. A lot of tricks have been developed to make this faster/more efficient. For example, AlphaGo uses a simpler classifier for rollouts than in the supervised learning layers. This results in rollout policies that are considerably less accurate than supervised learning policies, but, they are also considerably faster, so you can very quickly generate a ton of game simulations to evaluate a move.
The Second Edition of Sutton and Barto’s famous textbook on reinforcement learning has a full section just about rollout algorithms (8.10), and also more information on Monte Carlo sampling and Monte Carlo Tree Search (which has a strong simulation component). You can find a draft version here.
The definition of "rollouts" given by Planning chemical syntheses with deep neural networks and symbolic AI (Segler, Preuss & Waller ; doi: 10.1038/nature25978 ; credit to jsotola):
Rollouts are Monte Carlo simulations, in which random search steps are performed without branching until a solution has been found or a maximum depth is reached. These random steps can be sampled from machine-learned policies p(a|s), which predict the probability of taking the move (applying the transformation) a in position s, and are trained to predict the winning move by using human games or self-play
