Stereo Camera calibration with different camera types

Question

I'm trying to perform stereo camera calibration, rectification and disparity map generation. It's working fine with normal sample data. However, I'm trying to use the dual cameras on an iPhone 7+, which have different zoom. The telephoto lens has 2X zoom compared to the wide angle camera. I ran the images through the algorithm, and it is succeeding, although with a high error rate. However, when I open up the rectified images, they have a weird spherical look to the edges. The center looks fine. I'm assuming this is due to the cameras having different zoom levels. Is there anything special I need to do to deal with this? Or do I just need to crop any output to the usable undistorted area? Here is what I'm seeing:

EDIT:

I tried using the calibration result from these checkerboard images to rectify an image of some objects, and the rectification was way off, not even close. If I rectify one of my checkerboard images, they are spot on. Any ideas why that happens?

These are what my input images look like that result in the spherical looking output image. They are both taken from the exact same position, the iPhone was mounted to a tripod and I used a bluetooth device to trigger the shutter so the image wouldn't get shaken, my code automatically takes one image with each lens. I took 19 such images from different angles, all images show the full checkerboard. The more zoomed in image is the one that rectified to the top spherical looking image.

This is the code I am running. I compiled it and ran it using the sample images, that worked fine. Ran it with my own images and here I am. https://github.com/sourishg/stereo-calibration

I might just need to crop the result to a certain area. Regardless, it doesn't seem to work cropping or not when I use a picture I took of normal objects. Here is the output of an image of normal objects I ran through the filter:

Answer 1

It looks like what I saw when a colleague of mine was working with fisheye lenses; I found this post after doing an image search for "rectify fisheye image" and following the link from this picture.

At that post, they're using openCV to do the rectification. I'm always reluctant to bulk copy/paste someone else's content, so I'll just say that the tl;dr version of that post is:

Long story short, stereoCalibrate() couldn't handle the extreme distortion, but it you can remove that distortion before calling stereoCalibration().

As I asked in my comment above, you haven't stated what you're using to do the rectification, so I'm not sure how much this applies to you.

Hopefully this helps; if it doesn't, please edit your question to detail what exactly you're doing to generate that image.

Answer 2

Maybe you should rectify the wide-angle camera images with cv::fisheye::stereoRectify, which outputs a pair of undistort images with aligned epipolar lines.