I was trying to compare ORB-Slam2 performace with ORB-Slam3 performance. Consider following comparison from ORB-Slam3 paper (columns with numbers correspond to different sequences from EuRoC dataset): enter image description here

In 5 sequences (green), ORB-SLAM3 has better performance, in 4 (red), ORB-SLAM2 has better performance. One (blue) is tie. For one (pink), ORB-SLAM2 did not run completely, but its error is lesser in that case. Can we interpret this as ORB-SLAM-3 performs better in 50% of scenario and ORB-SLAM-2 in at least 40%.

I thought ORB-Slam3 should at least work considerably better than ORB-Slam2 at least when we use IMU. That is, running ORB-Slam3 in stereo inertial mode should perform better than ORB-Slam2 in stereo mode for same sequence / path trajectory. But as can be seen in below comparison, Stereo-ORB-Slam2 perform better (red underlined) than Stereo-inertial ORB-Slam3 in 5 sequences, while ORB-Slam3 performs better (green underlined) than Stereo-inertial ORB-Slam2 in 5 sequences. (In last sequence (pink underline), ORB-Slam2 did not run completely, ORB-Slam3 did run completely.)

enter image description here

Pointing out to discussions on different github issues:

  • Also, this comment says: ORB-SLAM3 was better but its trajectory is very jittery which is a big downside
  • This thread tries to discuss why ORB-SLAM2 perform better than ORB-SLAM3. Three people found ORB-SLAM2 performing better than ORB-SLAM3 (one is on KITTI dataset) with no contradiction.

So was guessing if ORB-Slam3 does really perform better than ORB-Slam2?

  • $\begingroup$ Not all environments are made the same and I don't believe its the case that the current SOTA in VSLAM has a single silver bullet method for all environments in all lighting conditions. In some places, some techniques are better than others. ORB SLAM 3 is clearly better, but that's not to say its perfect or that it always "considerably" outperforms other methods. $\endgroup$ Dec 21, 2023 at 18:51


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.