Problem understanding a paper about visual odometry

Question

I am reading the following paper: SVO: Semi-Direct Visual Odometry for Monocular and Multi-Camera Systems , and I am having some problem understanding some concepts.

Before starting I have to say that I am new to visual odometry, so maybe some simple concepts are not clear to me.

The chapter of the paper i cannot undesrtand is the one related to relaxation and refinement at pag.4 of the paper.

For what I have understood here, the paper says that for perfoeming image alignment, instead of using an image from the enviroment, it is better to use an image from a frame, so a 2D image instead of a 3D image, and in particular it is used the oldest frame in which the image was first reconstructed, and it is done because the oldest frame minimizes the feature drift (I don't undestand why).

Moreover, it says that by doing this, there is the problem of having a reprojection error.

After this, there is a more mathematical analysis, which is also really unclear to me.

I know that it is a lot to ask an explaination on a full chapter of a paper, but can somebody please help me?

Answer 1

So SVO works a bit differently then other VO systems as it uses dense image alignment. You need to understand this concept first before understanding SVO. Look up Lucas and Kanade image alignment. The best paper on this topic can be found here. This is required understanding, so you can't skip it.

Required Understanding:

• Lucas and Kanade(LK)
• How Sparse VO works (VINS MONO or ORB-SLAM2 are my recommendations)
• Bundle adjustment
• KLT tracking (OpenCV's KLT tracker)

Oldest Frame

The reason it uses the oldest frame is to eliminate the accumulation of errors. Imagine we have frames $F_1,F_2,F_3$. What VO does is estimate the transform $T_{ab}$ between 2 frames. Here our goal is to estimate the position of $T_{3}$. So $T_3$ is equal to $T_3=T_1*T_{12}*T_{23}$. Now notice here that we have estimate both $T_{12},T_{23}$. Every time we estimate we introduce error/drift into the system. Now what if instead we estimate $T_{1,3}$ directly. There is 1 less transform we need to estimate so 1 less chance for error.

Lets take a more extreme example with $F_1,...,F_{10}$. Which one will have less error?

$$ T_{10}=T_{1}*T_{12}*T_{23}*T_{34}*....T_{9,10}$$

or $$ T_{10}=T_{1}*T_{1,10}$$

Assuming it is possible to match $F_1$ and $F_{10}$.

2D vs 3D Image

Short answer:

3D dense image alignment does not work well when the images are far apart. So it switches to standard sparse VO and uses KLT tracking.

Longer answer:

It is possible to use the LK algorithm in multiple different ways depending on the warp. The way SVO uses it normally, and also how LSD-SLAM and DVO do it is through warping the 3D image with a 3D rigid body warp. A point we are warping is parameterized by $[u_{pixel},v_{pixel},d_{meters}]$(3D image). Now this formulation only works if the images are taken very close together. If they are too far apart then it switches to working like a Sparse VO system. It tracks the movement of the 2D keypoints in the image plane(KLT algorithm). Since we are tracking in the image plane we don't need the 3D information and can work only on the pixels. These 2D locations are then fed into the bundle adjustment algorithm.