I am trying to perform dense reconstruction using a sequence of images from a calibrated stereo camera. I have been using orb-slam3 to give me the camera's pose estimation. I am also generating the disparity map and RGB point cloud given a stereo image pair using available ROS packages. Suppose my sequence of stereo images is circling around a computer desk. Now I thought since I have the pose of the camera, point clouds representing the same region of the scene from different view points of the camera should overlap; however, it does not seem to be the case. The point cloud of the same object is shifted around depending on the location of the camera. I was hoping to get something similar to this:
So I was wondering if there is a component I'm still missing? Is having pose estimation and point clouds from a camera enough to generate a dense 3d map or do I still need to apply an algorithm like ICP?