Rosanswers logo

I'm new in this field and I'm trying to model a simple scene in 3d out of 2d images and I dont have any info about cameras. I know that there are 3 options ( ):

  • Or I have two images and I know the model of my camera (intrisics) that I loaded from a XML for instance loadXMLFromFile() => stereoRectify() => reprojectImageTo3D()

  • Or I don't have them but I can calibrate my camera => stereoCalibrate() => stereoRectify() => reprojectImageTo3D()

  • Or I can't calibrate the camera (it is my case, because I don't have the camera that has taken the 2 images, then : I need to find pair keypoints on both images with SURF, SIFT for instance (I can use any blob detector actually), then compute descriptors of these keypoints, then matching keypoints from image right and image left according to their descriptors, and then find the fundamental mat from them. The processing is much harder and would be like this:

  1. detect keypoints (SURF, SIFT) =>
  2. extract descriptors (SURF,SIFT) =>
  3. compare and match descriptors (BruteForce, Flann based approaches) =>
  4. find fundamental mat (findFundamentalMat()) from these pairs =>
  5. stereoRectifyUncalibrated() =>
  6. reprojectImageTo3D()

I'm using the last approach and my questions are:

  1. Is it right?

  2. if it's ok, I have a doubt about the last step "stereoRectifyUncalibrated() => reprojectImageTo3D()". The signature of reprojectImageTo3D() function is:

    void reprojectImageTo3D(InputArray disparity, OutputArray _3dImage, InputArray Q, bool handleMissingValues=false, int depth=-1 )

    cv::reprojectImageTo3D(imgDisparity8U, xyz, Q, true) (in my code)


-disparity – Input single-channel 8-bit unsigned, 16-bit signed, 32-bit signed or 32-bit floating-point disparity image. -_3dImage – Output 3-channel floating-point image of the same size as disparity . Each element of _3dImage(x,y) contains 3D coordinates of the point (x,y) computed from the disparity map. -Q – 4x4 perspective transformation matrix that can be obtained with stereoRectify(). handleMissingValues – Indicates, whether the function should handle missing values (i.e. points where the disparity was not computed). If handleMissingValues=true, then pixels with the minimal disparity that corresponds to the outliers (see StereoBM::operator() ) are transformed to 3D points with a very large Z value (currently set to 10000). -ddepth – The optional output array depth. If it is -1, the output image will have CV_32F depth. ddepth can also be set to CV_16S, CV_32S or CV_32F.

How can I get the Q matrix? Is possibile to obtain the Q matrix with F, H1 and H2 or in another way?

  1. Is there another way for obtain the xyz coordinates without calibrate the camera?

My code is:

#include <opencv2/core/core.hpp>
#include <opencv2/calib3d/calib3d.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/contrib/contrib.hpp>
#include <opencv2/features2d/features2d.hpp>
#include <stdio.h>
#include <iostream>
#include <vector>
#include <conio.h>
#include <opencv/cv.h>
#include <opencv/cxcore.h>
#include <opencv/cvaux.h>

using namespace cv;
using namespace std;

int main(int argc, char *argv[]){

    // Read the images
    Mat imgLeft = imread( argv[1], CV_LOAD_IMAGE_GRAYSCALE );
    Mat imgRight = imread( argv[2], CV_LOAD_IMAGE_GRAYSCALE );

    // check
    if (!imgLeft.data || !imgRight.data)
        return 0;

    // 1] find pair keypoints on both images (SURF, SIFT):::::::::::::::::::::::::::::

    // vector of keypoints
    std::vector<cv::KeyPoint> keypointsLeft;
    std::vector<cv::KeyPoint> keypointsRight;

   // Construct the SURF feature detector object
   cv::SiftFeatureDetector sift(
        0.01, // feature threshold
        10); // threshold to reduce
            // sensitivity to lines
            // Detect the SURF features

   // Detection of the SIFT features

   std::cout << "Number of SURF points (1): " << keypointsLeft.size() << std::endl;
   std::cout << "Number of SURF points (2): " << keypointsRight.size() << std::endl;

   // 2] compute descriptors of these keypoints (SURF,SIFT) ::::::::::::::::::::::::::

   // Construction of the SURF descriptor extractor
   cv::SurfDescriptorExtractor surfDesc;

   // Extraction of the SURF descriptors
   cv::Mat descriptorsLeft, descriptorsRight;

   std::cout << "descriptor matrix size: " << descriptorsLeft.rows << " by " <<   descriptorsLeft.cols << std::endl;

    // 3] matching keypoints from image right and image left according to their descriptors (BruteForce, Flann based approaches)

    // Construction of the matcher
    cv::BruteForceMatcher<cv::L2<float> > matcher;

    // Match the two image descriptors
    std::vector<cv::DMatch> matches;
    matcher.match(descriptorsLeft,descriptorsRight, matches);

    std::cout << "Number of matched points: " << matches.size() << std::endl;

    // 4] find the fundamental mat ::::::::::::::::::::::::::::::::::::::::::::::::::::

    // Convert 1 vector of keypoints into
    // 2 vectors of Point2f for compute F matrix
    // with cv::findFundamentalMat() function
    std::vector<int> pointIndexesLeft;
    std::vector<int> pointIndexesRight;
    for (std::vector<cv::DMatch>::const_iterator it= matches.begin(); it!= matches.end(); ++it) {

         // Get the indexes of the selected matched keypoints

    // Convert keypoints into Point2f
    std::vector<cv::Point2f> selPointsLeft, selPointsRight;

    /* check by drawing the points
    std::vector<cv::Point2f>::const_iterator it= selPointsLeft.begin();
    while (it!=selPointsLeft.end()) {

        // draw a circle at each corner location

    it= selPointsRight.begin();
    while (it!=selPointsRight.end()) {

        // draw a circle at each corner location
    } */

    // Compute F matrix from n>=8 matches
    cv::Mat fundemental= cv::findFundamentalMat(
        cv::Mat(selPointsLeft), // points in first image
        cv::Mat(selPointsRight), // points in second image
        CV_FM_RANSAC);       // 8-point method

    std::cout << "F-Matrix size= " << fundemental.rows << "," << fundemental.cols << std::endl;

    /* draw the left points corresponding epipolar lines in right image
    std::vector<cv::Vec3f> linesLeft;
        cv::Mat(selPointsLeft), // image points
        1,                      // in image 1 (can also be 2)
        fundemental,            // F matrix
        linesLeft);             // vector of epipolar lines

    // for all epipolar lines
    for (vector<cv::Vec3f>::const_iterator it= linesLeft.begin(); it!=linesLeft.end(); ++it) {

        // draw the epipolar line between first and last column

    // draw the left points corresponding epipolar lines in left image
    std::vector<cv::Vec3f> linesRight;
    for (vector<cv::Vec3f>::const_iterator it= linesRight.begin(); it!=linesRight.end(); ++it) {

        // draw the epipolar line between first and last column
        cv::line(imgLeft,cv::Point(0,-(*it)[2]/(*it)[1]), cv::Point(imgLeft.cols,-((*it)[2]+(*it)[0]*imgLeft.cols)/(*it)[1]), cv::Scalar(255,255,255));

    // Display the images with points and epipolar lines
    cv::namedWindow("Right Image Epilines");
    cv::imshow("Right Image Epilines",imgRight);
    cv::namedWindow("Left Image Epilines");
    cv::imshow("Left Image Epilines",imgLeft);

    // 5] stereoRectifyUncalibrated()::::::::::::::::::::::::::::::::::::::::::::::::::

    //H1, H2 – The output rectification homography matrices for the first and for the second images.
    cv::Mat H1(4,4, imgRight.type());
    cv::Mat H2(4,4, imgRight.type());
    cv::stereoRectifyUncalibrated(selPointsRight, selPointsLeft, fundemental, imgRight.size(), H1, H2);

    // create the image in which we will save our disparities
    Mat imgDisparity16S = Mat( imgLeft.rows, imgLeft.cols, CV_16S );
    Mat imgDisparity8U = Mat( imgLeft.rows, imgLeft.cols, CV_8UC1 );

    // Call the constructor for StereoBM
    int ndisparities = 16*5;      // < Range of disparity >
    int SADWindowSize = 5;        // < Size of the block window > Must be odd. Is the 
                              // size of averaging window used to match pixel  
                              // blocks(larger values mean better robustness to
                              // noise, but yield blurry disparity maps)

    StereoBM sbm( StereoBM::BASIC_PRESET,
       SADWindowSize );

    // Calculate the disparity image
    sbm( imgLeft, imgRight, imgDisparity16S, CV_16S );

    // Check its extreme values
    double minVal; double maxVal;

    minMaxLoc( imgDisparity16S, &minVal, &maxVal );

    printf("Min disp: %f Max value: %f \n", minVal, maxVal);

    // Display it as a CV_8UC1 image
    imgDisparity16S.convertTo( imgDisparity8U, CV_8UC1, 255/(maxVal - minVal));

    namedWindow( "windowDisparity", CV_WINDOW_NORMAL );
    imshow( "windowDisparity", imgDisparity8U );

    // 6] reprojectImageTo3D() :::::::::::::::::::::::::::::::::::::::::::::::::::::

    //Mat xyz;
    //cv::reprojectImageTo3D(imgDisparity8U, xyz, Q, true);

    //How can I get the Q matrix? Is possibile to obtain the Q matrix with 
    //F, H1 and H2 or in another way?
    //Is there another way for obtain the xyz coordinates?

    return 0;

best regards


Originally posted by Fobi on ROS Answers with karma: 11 on 2012-01-27

Post score: 1


1 Answer 1


Rosanswers logo

Why are you mixing opencv and opencv2 headers? I tried this code with a couple of test images and some did ok and some were bad. Maybe play with the parameters some to get better performance.

Since this is really an OpenCV question, you might want to ask it on their discussion board.

Originally posted by Kevin with karma: 2962 on 2012-02-10

This answer was ACCEPTED on the original site

Post score: 1


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.