The standard use of “rollout” (also called a “playout”) is in regard to an execution of a policy from the current state when there is some uncertainty about the next state or outcome - it is one simulation from your current state. The purpose is for an agent to evaluate many possible next actions in order to find an action that will maximize value (long-term expected reward).
Uncertainty in the next state can arise from different sources depending on your domain. In games the uncertainty is typically from your opponent (you are not certain what move they will make next) or a chance element (e.g. a dice roll). In robotics, you may be modeling uncertainty in your environment (e.g. your perception system gives inaccurate pose estimates, so you are not sure an object is where you think it is) or your robot (e.g. noisy sensors result in unreliable transition dynamics).
I think the term comes from Tesauro and Galperin NIPS 1997 in which they consider Monte Carlo simulations of Backgammon where a playout considers a sequence of dice rolls:
In backgammon parlance, the expected value of a position is known as
the "equity" of the position, and estimating the equity by Monte-Carlo
sampling is known as performing a "rollout." This involves playing the
position out to completion many times with different random dice
sequences, using a fixed policy P to make move decisions for both
As you could imagine, this can be quite computationally expensive. A lot of tricks have been developed to make this faster/more efficient. For example, AlphaGo uses a simpler classifier for rollouts than in the supervised learning layers. This results in rollout policies that are considerably less accurate than supervised learning policies, but, they are also considerably faster, so you can very quickly generate a ton of game simulations to evaluate a move.
The Second Edition of Sutton and Barto’s famous textbook on reinforcement learning has a full section just about rollout algorithms (8.10), and also more information on Monte Carlo sampling and Monte Carlo Tree Search (which has a strong simulation component). You can find a draft version here.