What are myopic and non-myopic policies?

Question

I am reading this paper: A survey on Policy Search for Robotics and I came across these terms. Can someone give an answer, please?

Answer 1

A myopic policy is one that simply maximises the average immediate reward. It is "myopic" in the sense that it only considers the single criterion. It has the advantage of being relatively easy to implement. A fairly well-known example is the hill-climbing algorithm.

However, a myopic search is particularly vulnerable to becoming trapped at a local optima, and so failing to identify a globally optimal solution.

---

A non-myopic policy is one which has wider focus, considering other factors. This can greatly improve the success of the algorithm, but is generally more complex to implement.

Answer 2

You first have to be clear about the core RL terms, to understand myopic and non-myopic policies:

• Policy: Suppose each cell in the grid below is a state that the RL agent can be in. From each state, it can transition to another state. Now, the arrows that are given, shows a particular transition it would take from each state to go to another. This depicts a policy that answers (Which direction should I go from any current state?)
  
• Reward: For every transition or a sequence of transitions, the agent gets a Reward for it. It can be positive or negative.

Now, in Reinforcement learning, an agent tries to pick a policy that gives it the maximum reward in the future. The agent might just look 1 step ahead in the future or 1000 (an arbitrary number) steps. If it looks 1 step ahead and based on that picks a policy that gives it a maximum reward, the policy would be myopic. On the other hand, if the agent looks 1000 steps ahead in future and then decides on the best policy, the policy would be termed non-myopic.

In RL, most generally, the number of steps to look ahead in future is determined by the discounting factor $\gamma \in [0,1]$, when the total reward $G_t$ is formulated as:

$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} ... = \sum_{k=0}^{\infty}\gamma^k R_{t+k+1}$

As you can see, if $\gamma = 0$, the total reward $G_t$ will contain reward of only one time-step ahead in future $R_{t+1}$ (myopic). If $\gamma = 1$, rewards till $\infty$ are considered (non-myopic).

You can follow Dr. David Silver's lecture on RL to learn more: Link