Pipeline for dense reconstruction using pose estimation from orb-slam and stereo camera

Question

I am trying to perform dense reconstruction using a sequence of images from a calibrated stereo camera. I have been using orb-slam3 to give me the camera's pose estimation. I am also generating the disparity map and RGB point cloud given a stereo image pair using available ROS packages. Suppose my sequence of stereo images is circling around a computer desk. Now I thought since I have the pose of the camera, point clouds representing the same region of the scene from different view points of the camera should overlap; however, it does not seem to be the case. The point cloud of the same object is shifted around depending on the location of the camera. I was hoping to get something similar to this:

So I was wondering if there is a component I'm still missing? Is having pose estimation and point clouds from a camera enough to generate a dense 3d map or do I still need to apply an algorithm like ICP?

Answer 1

Small errors and outliers in camera poses will make the reconstruction unprecise, hence you need some refinement.

You have multiple options:

• Fuse the point clouds with an ICP-like algorithm.
• Use a photogrammetry technique.
• Use neural synthesis techniques such as this or this.