15
$\begingroup$

I am considering programming a line following robot using reinforcement learning algorithms. The question I am pondering over is how can I get the algorithm to learn navigating through any arbitrary path?

Having followed the Sutton & Barto Book for reinforcement learning, I did solve an exercise problem involving a racetrack where in the car agent learnt not to go off the track and regulate its speed. However, that exercise problem got the agent to learn how to navigate the track it trained on.

Is it in the scope of reinforcement learning to get a robot to navigate arbitrary paths? Does the agent absolutely have to have a map of the race circuit or path? What parameters could I possibly use for my state space?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ I don't know how, but I'm certain that it is possible to teach it to keep itself within a path, regardless of its shape. The goal of teaching in this case should be what local decision the robot takes based on its immediate inputs (or perhaps some history). This way, it doesn't matter what the shape of the path is, since all decisions are local. $\endgroup$
    – Shahbaz
    Nov 11, 2012 at 20:48
  • $\begingroup$ @Shahbaz - What would you suggest I use for the state space? $\endgroup$
    – Lord Loh.
    Nov 12, 2012 at 18:42
  • $\begingroup$ I'm actually not sure. Even though I have studied AI in university, that was never my field of work/research. The answer you have accepted seems reasonable! $\endgroup$
    – Shahbaz
    Nov 12, 2012 at 18:58
  • $\begingroup$ Are you using an answer to test Markdown? You can just write what you want and see the immediate rendering below it, and then not post it. $\endgroup$
    – Shahbaz
    Nov 14, 2014 at 10:11

2 Answers 2

Reset to default
10
$\begingroup$

One of the key measures of any machine learning algorithm is it's ability to generalize (i.e. apply what it has learned to previously unsceen scenarios). Reinforcement learners (RL) can generalize well but this ability is in part a function of the state-space formulation in my experience. This means that if you can find the right setup then the RL learner will not need a map of of the race circuit.

This leaves the question of which parameters to use. Without knowing more about the sensors available on your robot I can only speculate. My first inclinition is to try to encoded the relative orientation of the line and robot (i.e. is the robot tending to the right, left, or simply moving parallel with the line). Doing so would result in a nice small state-space. Though not strictly necessary it would make for a quick and simple implementation. Furthermore, if the robot is not going to move at a constant rate then it may help to encode the robots velocity as the robot will need to react more quickly when moving at higher speeds.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Without the map, the state space is just what the robot can sense from its immediate position. So to a certain extent, the map is just a way to "look ahead". The learned behavior in the map-less scenario will be essentially "do the same thing as before but go slower because we don't know where the turns are this time". (Presumably, you'll be able to tell where the edges of the track are.) $\endgroup$
    – Ian
    Aug 27, 2013 at 18:40
  • $\begingroup$ A policy tells us what control to apply given a state. If the state-space is formulated well for the RL agent then distinctly different scenarios may look the same in the state-space and yield the same behavior. This is called generalization and is desirable when done correctly. The robot will have a maximum speed based on the speed of it's control loop. The learned behavior will not necessarily be to slow down. If the reward is inversely related to the length of time of the run then the agent would be inclined to max out it's speed with respect to the speed of it's control loop. $\endgroup$
    – DaemonMaker
    Aug 27, 2013 at 19:07
-1
$\begingroup$

I am not sure what type of robot you have but i have been doing robocup rescue line for a few years now. I have come to realise that if you want to follow a line well using PID is a good option. Let me expand on this. If you imagine two light sensors on either side of the line you would want them to be of equal value so that the line is in the middle. You can then use the difference between the value of the two sensors to change the turning percentage of the robot. With this technique it is possible to get a robot to follow a line at extraordinary speeds.

In terms of making the robot learn to improve its line tracking abilities. What i came up with is start with your initial PID values higher than you want them to be and use a gyroscopic sensor to measure the frequency of oscillation of the robot as it tracks the line. You can from there create your own function to determine how much to lower your values which acts as an automated reinforcement learning parameter. So if the oscillations are too high or become unstable then you state this set of parameters are bad.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.