Skip to main content
Robotics

Return to Answer

deleted 12 characters in body
Source Link
ricecakebear
ricecakebear
  • 151
  • 6
// ....
//solve calibration

//generate a 3-channel mat with each entry containing it's own coordinates
Mat xyz(2000, 2500, CV_32FC3);
float *pxyz = (float*)xyz.data;
for (int y = 0; y < 2000; y++)
    for (int x = 0; x < 2500; x++)
    {
        *pxyz++ = x;
        *pxyz++ = y;
        *pxyz++ = 0;
    }

// project coordinates of destination image,
// which generates the map from destination image to source image directly
xyz=xyz.reshape(0, 5000000);
Mat mapToSrcImgmapToSrc(5000000, 1, CV_32FC2);
projectPoints(xyz, rvecs[0], tvecs[0], intrinsic, distCoeffs, mapToSrcImgmapToSrc);
Mat maps[2];
mapToSrcImgmapToSrc = pts_destmapToSrc.reshape(0, 2000);
split(mapToSrcImgmapToSrc, maps);

//apply map
remap(imgB, imgC, maps[0], maps[1], INTER_LINEAR);

// ....
//solve calibration

//generate a 3-channel mat with each entry containing it's own coordinates
Mat xyz(2000, 2500, CV_32FC3);
float *pxyz = (float*)xyz.data;
for (int y = 0; y < 2000; y++)
    for (int x = 0; x < 2500; x++)
    {
        *pxyz++ = x;
        *pxyz++ = y;
        *pxyz++ = 0;
    }

// project coordinates of destination image,
// which generates the map from destination image to source image directly
xyz=xyz.reshape(0, 5000000);
Mat mapToSrcImg(5000000, 1, CV_32FC2);
projectPoints(xyz, rvecs[0], tvecs[0], intrinsic, distCoeffs, mapToSrcImg);
Mat maps[2];
mapToSrcImg = pts_dest.reshape(0, 2000);
split(mapToSrcImg, maps);

//apply map
remap(imgB, imgC, maps[0], maps[1], INTER_LINEAR);

// ....
//solve calibration

//generate a 3-channel mat with each entry containing it's own coordinates
Mat xyz(2000, 2500, CV_32FC3);
float *pxyz = (float*)xyz.data;
for (int y = 0; y < 2000; y++)
    for (int x = 0; x < 2500; x++)
    {
        *pxyz++ = x;
        *pxyz++ = y;
        *pxyz++ = 0;
    }

// project coordinates of destination image,
// which generates the map from destination image to source image directly
xyz=xyz.reshape(0, 5000000);
Mat mapToSrc(5000000, 1, CV_32FC2);
projectPoints(xyz, rvecs[0], tvecs[0], intrinsic, distCoeffs, mapToSrc);
Mat maps[2];
mapToSrc = mapToSrc.reshape(0, 2000);
split(mapToSrc, maps);

//apply map
remap(imgB, imgC, maps[0], maps[1], INTER_LINEAR);
added 1235 characters in body
Source Link
ricecakebear
ricecakebear
  • 151
  • 6

Added 3

I realized that as each frame captured by camera needs processing and efficiency is important, I can't afford to use undistort and warpPerspective for each frame. It's only reasonable to have one map and use remap for each frame.

Actually, there is a straight forward way to that, which is projectPoints. Since it generates the map from destination image to source image directly, no intermediary image is needed and thus loss of information is avoided.

// ....
//solve calibration

//generate a 3-channel mat with each entry containing it's own coordinates
Mat xyz(2000, 2500, CV_32FC3);
float *pxyz = (float*)xyz.data;
for (int y = 0; y < 2000; y++)
    for (int x = 0; x < 2500; x++)
    {
        *pxyz++ = x;
        *pxyz++ = y;
        *pxyz++ = 0;
    }

// project coordinates of destination image,
// which generates the map from destination image to source image directly
xyz=xyz.reshape(0, 5000000);
Mat mapToSrcImg(5000000, 1, CV_32FC2);
projectPoints(xyz, rvecs[0], tvecs[0], intrinsic, distCoeffs, mapToSrcImg);
Mat maps[2];
mapToSrcImg = pts_dest.reshape(0, 2000);
split(mapToSrcImg, maps);

//apply map
remap(imgB, imgC, maps[0], maps[1], INTER_LINEAR);

Added 3

I realized that as each frame captured by camera needs processing and efficiency is important, I can't afford to use undistort and warpPerspective for each frame. It's only reasonable to have one map and use remap for each frame.

Actually, there is a straight forward way to that, which is projectPoints. Since it generates the map from destination image to source image directly, no intermediary image is needed and thus loss of information is avoided.

// ....
//solve calibration

//generate a 3-channel mat with each entry containing it's own coordinates
Mat xyz(2000, 2500, CV_32FC3);
float *pxyz = (float*)xyz.data;
for (int y = 0; y < 2000; y++)
    for (int x = 0; x < 2500; x++)
    {
        *pxyz++ = x;
        *pxyz++ = y;
        *pxyz++ = 0;
    }

// project coordinates of destination image,
// which generates the map from destination image to source image directly
xyz=xyz.reshape(0, 5000000);
Mat mapToSrcImg(5000000, 1, CV_32FC2);
projectPoints(xyz, rvecs[0], tvecs[0], intrinsic, distCoeffs, mapToSrcImg);
Mat maps[2];
mapToSrcImg = pts_dest.reshape(0, 2000);
split(mapToSrcImg, maps);

//apply map
remap(imgB, imgC, maps[0], maps[1], INTER_LINEAR);
Source Link
ricecakebear
ricecakebear
  • 151
  • 6

Added

OK, guys, simple mistake. I previously used warpPerspective to warp images instead of restoring. Since it works that way, I didn't read the doc thoroughly. It turns out that if it is for restoring, the flag WARP_INVERSE_MAP should be set. Change the function call to this, and that's it.

warpPerspective(tempImgC, imgC, matPerspective, Size(2500, 2000), WARP_INVERSE_MAP);

Here is the new result image C: enter image description here

The only thing concerns me now is the intermediary tempImgC, which is the image after undistort and before warpPerspective. In some tests with different artificial B s, This image could turn out to be a scaled-up version of B with distortion removed. That means a lot of information is lost in the outter area. And there is not much to use for warpPerspective. I'm thinking maybe to scale down the image in undistort and to scale it up in warpPerspective. But I'm not sure yet how to calculate the correct scale to preserve all the information in B.

Added 2

The last piece of the puzzle is in place. Call getOptimalNewCameraMatrix before undistort to generate the new camera matrix that preserves all the info in B. And pass this new camera matrix to undistort and warpPerspective.

Mat newIntrinsic=getOptimalNewCameraMatrix(intrinsic, distCoeffs, Size(2500, 2000), 1);
undistort(imgB, tempImgC, intrinsic, distCoeffs, newIntrinsic);
Mat matPerspective = newIntrinsic*transRot3x3;
warpPerspective(tempImgC, imgC, matPerspective, Size(2500, 2000), WARP_INVERSE_MAP);

The result image C is the same in this case. But there is a big difference for other cases. For example, with another distorted image B1. enter image description here The result image C1 without new camera matrix looks like this. enter image description here And the result image C1 with new camera matrix maintains the information in B1 enter image description here