7
$\begingroup$

I have 2 wheeled differential drive robot which I use pid for low level control to follow line. I implemented q learning which uses samples for 16 iterations then uses them to decide the best position to be on the line so car takes the turn from there. This allows PID to setup and smooth fast following. My question is how can I setup a reward function that improves the performance i.e. lets the q learning to find the best

Edit

What it tries to learn is this, it has 16 inputs which contains the line positions for the last 15 iterations and this iteration. Line position is between -1 and 1 which -1 means only left most sensor sees the line and 0 means the line is in the center. I want it to learn a line position that when it faces this input again it will set that line position like its the center and take the curve according to that line position. For example error is required position - line position so let say I had 16 0 as input then I calculated the required as 0.4. So after that the car will center itself at 0.4 I hope this helps :)

You asked for my source code i post it below

void MainController::Control(void){

    float linePosition = sensors->ReadSensors();

    if(linePosition == -2.0f){       
      lost_line->FindLine(lastPos[1] - lastPos[0]);
    } 
    else{
        line_follower->Follow(linePosition);
        lastPos.push_back(linePosition);
        lastPos.erase(lastPos.begin());   
    }
}

My Sensor reading returns a value between -1.0f and 1.0f. 1.0f means Outer Sensor on the right is only the line. I have 8 sensors.

void LineFollower::Follow(float LinePosition){

    float requiredPos = Qpredictor.Process(LinePosition,CurrentSpeed);
    float error = requiredPos - LinePosition;

    float ErrorDer = error -LastError;

    float diffSpeed = (KpTerm * error + (KdTerm * ErrorDer));

    float RightMotorSpeed = CurrentSpeed - diffSpeed;
    float LeftMotorSpeed = CurrentSpeed + diffSpeed;

    LastError = error;

    driver->Drive(LeftMotorSpeed,RightMotorSpeed);
}

Here is the logic for the value for QPredictor(I call the learning part as this). And Finally QPredictor

float Memory[MemorySize][DataVectorLength] =
{
  {0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0},
  {0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3, 0.3},
  {0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6, 0.6},
  {0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8, 0.8},

  {0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.012, 0.050, 0.113, 0.200, 0.312, 0.450, 0.613, 0.800, 1.000},

  {0.000, 0.025, 0.100, 0.225, 0.400, 0.625, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},

  {0.000, 0.050, 0.200, 0.450, 0.800, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},


  {0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000, 1.000},
  {0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.000, 0.100, 0.400, 0.900, 1.000}
};


QPredictor::QPredictor(){
    for(int i=0;i<MemorySize;i++){
        output[i]=0.0f;
        input[i]=0.0f;
    }

    state = 0;
    PrevState = 0;
}

float QPredictor::Process(float linePosition,float currentBaseSpeed){
    for(int i=1;i<DataVectorLength;i++){
        input[i] = input[i-1];
    }

    input[0] = m_abs(linePosition);

    int MinIndex = 0;
    float Distance = 10000.0f;

    float sum = 0.0f;

    for(int i=0;i<MemorySize;i++){
        sum = 0.0f;
        for(int j=0;j<DataVectorLength;j++){
            sum +=m_abs(input[j] - Memory[i][j]);
        }

        if(sum <= Distance){
            MinIndex = i;
            Distance = sum;     
       }
    }

    sum = 0.0f;
    for(int i=0;i<DataVectorLength;i++){
        sum += input[i];
    }

    float eta = 0.95f;

    output[MinIndex] = eta * output[MinIndex] + (1 - eta) * sum;

    return -m_sgn(linePosition) * output[MinIndex];
}


float QPredictor::rewardFunction(float *inputData,float currentBaseSpeed){
    float sum = 0.0f;

    for(int i=0;i<DataVectorLength;i++){
        sum += inputData[i];
    }

    sum /= DataVectorLength;

    return sum;
}

I now only have average Error and currently not using learning because it's not complete without reward function. How can I adjust it according to the dimensions of my Robot?

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ I think your question needs some improvement before it can be answered. E.g. what are you trying to learn? The PID parameters? What is your goal for the final systems behaviour? $\endgroup$
    – Jakob
    Aug 15, 2016 at 6:32
  • $\begingroup$ linked question: robotics.stackexchange.com/questions/361/… $\endgroup$
    – Manuel Rodriguez
    Aug 15, 2016 at 8:56
  • $\begingroup$ The memory-array stores the past experience of robot, not the future. Negative rewards must given if all array elements are white. For simple curves this will work. $\endgroup$
    – Manuel Rodriguez
    Aug 18, 2016 at 21:50
  • $\begingroup$ Yes that makes sense but what if i havent lost line how can i increase or decrease the reward or punishment $\endgroup$
    – Ege Keyvan
    Aug 19, 2016 at 6:24
  • $\begingroup$ +1 for attempt; You can use a simple webcam for validation; Use simple check to determine how close your robot is to the centre of the line. Then, use that and add rewards (like +1 indicating "continue on the lane" every second) and punishments (if one of the wheels cross the line; -50 for every second). $\endgroup$
    – Prasad Raghavendra
    Apr 9, 2017 at 12:37

2 Answers 2

Reset to default
0
$\begingroup$

Adaptive PID control is based on a manual coded algorithm for line-following which is modified by reinforcement learning. The idea is, that in the PID function a parameter is unknown (e.g. the distance from line which can be tolerated as deviation) and this parameter will be calculated on-the-fly while driving the robot. The rewards are given manual by an operator Clicker training or can be determined in a loop. The robot is driving a parcurs, the time is measured. The robot drives the parcurs again and the goal is to decrease the time.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I have seen it but i have kohonen maps and q learning that i use kohonen to cluster the input vector and use q learning to decide the output can i adapt this system to mine ? $\endgroup$
    – Ege Keyvan
    Aug 15, 2016 at 9:07
  • $\begingroup$ Sure, the self-organizing map can be used as associative memory (Q-KOHON, Claude F. Touzet). $\endgroup$
    – Manuel Rodriguez
    Aug 15, 2016 at 9:18
  • $\begingroup$ Do you suggest according to the input should the system change the parameters ? $\endgroup$
    – Ege Keyvan
    Aug 15, 2016 at 9:45
  • $\begingroup$ @EgeKeyvan: please post your soucecode or a screenshot of NXT-G GUI $\endgroup$
    – Manuel Rodriguez
    Aug 15, 2016 at 14:48
  • $\begingroup$ I posted it now $\endgroup$
    – Ege Keyvan
    Aug 17, 2016 at 12:32
0
$\begingroup$

I think Q-learning is an overkill for such a simple tracking problem. I highly recommend you using a simpler method, namely pure pursuit tracking controller. Here are some links that you can read about the method. Matlab's page on the subject is pretty neat.

Matlab's Pure Pursuit Controller

Implementation of the Pure Pursuit Path Tracking Algorithm

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.