At the moment I try to understand Bundle Adjustment on a level to be able to implement it on my own. The major point I'm still confused at is the management of the landmarks. So, let's think of a few frames, that all share some landmarks with each other (like in the image below).

Stolen from: SBA: A Software Package for Generic Sparse Bundle Adjustment, MANOLIS I. A. LOURAKIS and ANTONIS A. ARGYROS

I can triangulate the same corresponding feature in image 1 to image 2, from image 2 to image 3 and so forth. However, I get (slightly) different results for the same point in the world. I first tried it with fusing all the landmark estimates from the different triangulations (see an older post). But, I'm not sure if that is really the way to go.

If my question is still a bit to unclear, please let me know and I will improve it. I'm looking forward to interesting answers.


First thing to do is pick your World frame. This is your main frame where the 3D coordinates of your points and poses of your cameras will be. Typically the origin of this world frame is the pose of your first camera. You then triangulate every point with all of the cameras that see it. A rough guess can be computed with an algorithm like DLT, and then refined with some non-linear optimization. Some source code for this can be found here(notice it has from N views triangulate). Make sure the output for this is actually in the world frame and not relative to some camera.

Now put all of your camera poses and 3D points in a bundle adjustment problem. You have 2 sets of variables. Each landmark is going to be connected to every camera pose that views it via the reprojection error. Solve it using a non-linear least squares solver like Ceres. They even have a basic tutorial with some given 3D points and camera poses, and reprojections.

  • $\begingroup$ Thank you for your answer. That sounds like a good way to go. Maybe I should have mentioned, that I look more in the online-windowed direction (my bad). However, may it be a reasonable way to just triangulate from the first two views and then use this landmark for all upcoming frames in the window? $\endgroup$ – MattDom Nov 27 '19 at 17:38
  • 1
    $\begingroup$ Yes that is how you would do it. Though usually I would wait until there are 3 views of the landmark before triangulating it and adding it to the BA problem. In the online approach you also want to think about keyframing. So you only track and triangulate points that occur in a specified Keyframe. $\endgroup$ – edwinem Nov 28 '19 at 5:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.