# Reward Function for q learning on a robot

I have 2 wheeled differential drive robot which I use pid for low level control to follow line. I implemented q learning which uses samples for 16 iterations then uses them to decide the best position to be on the line so car takes the turn from there. This allows PID to setup and smooth fast following. My question is how can I setup a reward function that improves the performance i.e. lets the q learning to find the best

Edit

What it tries to learn is this, it has 16 inputs which contains the line positions for the last 15 iterations and this iteration. Line position is between -1 and 1 which -1 means only left most sensor sees the line and 0 means the line is in the center. I want it to learn a line position that when it faces this input again it will set that line position like its the center and take the curve according to that line position. For example error is required position - line position so let say I had 16 0 as input then I calculated the required as 0.4. So after that the car will center itself at 0.4 I hope this helps :)

You asked for my source code i post it below

void MainController::Control(void){

if(linePosition == -2.0f){
lost_line->FindLine(lastPos[1] - lastPos[0]);
}
else{
line_follower->Follow(linePosition);
lastPos.push_back(linePosition);
lastPos.erase(lastPos.begin());
}
}


My Sensor reading returns a value between -1.0f and 1.0f. 1.0f means Outer Sensor on the right is only the line. I have 8 sensors.

void LineFollower::Follow(float LinePosition){

float requiredPos = Qpredictor.Process(LinePosition,CurrentSpeed);
float error = requiredPos - LinePosition;

float ErrorDer = error -LastError;

float diffSpeed = (KpTerm * error + (KdTerm * ErrorDer));

float RightMotorSpeed = CurrentSpeed - diffSpeed;
float LeftMotorSpeed = CurrentSpeed + diffSpeed;

LastError = error;

driver->Drive(LeftMotorSpeed,RightMotorSpeed);
}


Here is the logic for the value for QPredictor(I call the learning part as this). And Finally QPredictor

float Memory[MemorySize][DataVectorLength] =
{
{0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0},
{0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3},
{0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6},
{0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8},

{0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000},

{0.000, 0.025, 0.100, 0.225, 0.400, 0.625, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},

{0.000, 0.050, 0.200, 0.450, 0.800, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},

{0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000},
{0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000}
};

QPredictor::QPredictor(){
for(int i=0;i<MemorySize;i++){
output[i]=0.0f;
input[i]=0.0f;
}

state = 0;
PrevState = 0;
}

float QPredictor::Process(float linePosition,float currentBaseSpeed){
for(int i=1;i<DataVectorLength;i++){
input[i] = input[i-1];
}

input[0] = m_abs(linePosition);

int MinIndex = 0;
float Distance = 10000.0f;

float sum = 0.0f;

for(int i=0;i<MemorySize;i++){
sum = 0.0f;
for(int j=0;j<DataVectorLength;j++){
sum +=m_abs(input[j] - Memory[i][j]);
}

if(sum <= Distance){
MinIndex = i;
Distance = sum;
}
}

sum = 0.0f;
for(int i=0;i<DataVectorLength;i++){
sum += input[i];
}

float eta = 0.95f;

output[MinIndex] = eta * output[MinIndex] + (1 - eta) * sum;

return -m_sgn(linePosition) * output[MinIndex];
}

float QPredictor::rewardFunction(float *inputData,float currentBaseSpeed){
float sum = 0.0f;

for(int i=0;i<DataVectorLength;i++){
sum += inputData[i];
}

sum /= DataVectorLength;

return sum;
}


I now only have average Error and currently not using learning because it's not complete without reward function. How can I adjust it according to the dimensions of my Robot?

• I think your question needs some improvement before it can be answered. E.g. what are you trying to learn? The PID parameters? What is your goal for the final systems behaviour? – Jakob Aug 15 '16 at 6:32
• linked question: robotics.stackexchange.com/questions/361/… – Manuel Rodriguez Aug 15 '16 at 8:56
• The memory-array stores the past experience of robot, not the future. Negative rewards must given if all array elements are white. For simple curves this will work. – Manuel Rodriguez Aug 18 '16 at 21:50
• Yes that makes sense but what if i havent lost line how can i increase or decrease the reward or punishment – Ege Keyvan Aug 19 '16 at 6:24
• +1 for attempt; You can use a simple webcam for validation; Use simple check to determine how close your robot is to the centre of the line. Then, use that and add rewards (like +1 indicating "continue on the lane" every second) and punishments (if one of the wheels cross the line; -50 for every second). – Prasad Raghavendra Apr 9 '17 at 12:37

Adaptive PID control is based on a manual coded algorithm for line-following which is modified by reinforcement learning. The idea is, that in the PID function a parameter is unknown (e.g. the distance from line which can be tolerated as deviation) and this parameter will be calculated on-the-fly while driving the robot. The rewards are given manual by an operator Clicker training or can be determined in a loop. The robot is driving a parcurs, the time is measured. The robot drives the parcurs again and the goal is to decrease the time.

• I have seen it but i have kohonen maps and q learning that i use kohonen to cluster the input vector and use q learning to decide the output can i adapt this system to mine ? – Ege Keyvan Aug 15 '16 at 9:07
• Sure, the self-organizing map can be used as associative memory (Q-KOHON, Claude F. Touzet). – Manuel Rodriguez Aug 15 '16 at 9:18
• Do you suggest according to the input should the system change the parameters ? – Ege Keyvan Aug 15 '16 at 9:45
• @EgeKeyvan: please post your soucecode or a screenshot of NXT-G GUI – Manuel Rodriguez Aug 15 '16 at 14:48
• I posted it now – Ege Keyvan Aug 17 '16 at 12:32

I think Q-learning is an overkill for such a simple tracking problem. I highly recommend you using a simpler method, namely pure pursuit tracking controller. Here are some links that you can read about the method. Matlab's page on the subject is pretty neat.

Matlab's Pure Pursuit Controller

Implementation of the Pure Pursuit Path Tracking Algorithm