cv2.stereoRectify works only when the rotation and translation are from camera 2 to camera 1

Question

I am using the euroc-mav dataset to create a disparity map from stereo images: https://projects.asl.ethz.ch/datasets/doku.php?id=kmavvisualinertialdatasets.

In this dataset the cameras are already calibrated (intrinsic and extrinsic) relative to a common coordinate system (the imu frame).

My goal is to rectify both of the images to create a disparity map. I am using the opencv cv2.stereoRectify().

On the opencv documentation, the stereoRectify function receive the rotation and translation between the two cameras (from the coordinate system of camera 1 to the coordinate system of camera 2).

I first calculated the rotation between as follow:

relative_R_cam0_cam1 = np.linalg.inv(R_cam0) @ R_cam1
relative_T_cam0_cam1 = np.linalg.inv(R_cam0) @ (T_cam1 - T_cam0) 

With this I get the disparity map to look very bad.

I then check what happened when I calculate the rotation and translation from camera 2 frame to camera 1 frame with the following change:

relative_R_cam0_cam1 = np.linalg.inv(R_cam1) @ R_cam0
relative_T_cam0_cam1 = np.linalg.inv(R_cam1) @ (T_cam0 - T_cam1)

I received a good disparity map:

Is there a bug in the opencv stereoRectify ? or I am missing something?

Here is my entire code:

cam0_intrinsics = np.array([
    [458.654, 0.0, 367.215],
    [0.0, 457.296 , 248.3759],
    [0.0, 0.0, 1.0]])
cam0_distortion = np.array([[-0.28340811, 0.07395907, 0.00019359, 1.76187114e-05]])


cam1_intrinsics = np.array([
    [457.587, 0.0, 379.999],
    [0.0, 456.134 , 255.238],
    [0.0, 0.0, 1.0]])
cam1_distortion = np.array([[-0.28368365, 0.07451284, -0.00010473, -3.55590700e-05]])

R_cam0 = np.array([
    [0.0148655429818, -0.999880929698, 0.00414029679422],
    [0.999557249008, 0.0149672133247, 0.025715529948],
    [-0.0257744366974, 0.00375618835797, 0.999660727178]
])

T_cam0 = np.array([-0.0216401454975, -0.064676986768, 0.00981073058949])


R_cam1 = np.array([
    [0.0125552670891, -0.999755099723, 0.0182237714554],
    [0.999598781151, 0.0130119051815, 0.0251588363115],
    [-0.0253898008918, 0.0179005838253, 0.999517347078]
])

T_cam1 = np.array([-0.0198435579556, 0.0453689425024, 0.00786212447038])


relative_R_cam0_cam1 = np.linalg.inv(R_cam1) @ R_cam0
relative_T_cam0_cam1 = np.linalg.inv(R_cam1) @ (T_cam0 - T_cam1) 
# relative_R_cam0_cam1 = np.linalg.inv(R_cam0) @ R_cam1
# relative_T_cam0_cam1 = np.linalg.inv(R_cam0) @ (T_cam1 - T_cam0) 



R1, R2, P1, P2, Q, roi1, roi2 = cv2.stereoRectify( \
        cam0_intrinsics, cam0_distortion, cam1_intrinsics, cam1_distortion, (752, 480), relative_R_cam0_cam1, relative_T_cam0_cam1)