Understanding reprojection error / visual residual in ORB-SLAM3

Question

I am trying to understand ORB-SLAM3 paper. It gives visual residual / reprojectio error equation as follows:

$$r_{ij}=u_{ij}-\prod(T_{CB}T^{-1}_i\oplus x_j)$$ where,

• $r_{ij}$ is reprojection error between frame $i$ and 3D point $j$ at point $x_j$
• $\prod:\mathbb{R}^3\rightarrow\mathbb{R}^n$ is the projecion function for the corresponding camera model
• $u_{ij}$ is the obeservaton of point $j$ at image $i$ having a covariance matrix $\sum_{ij}$
• $T_{CB}\in SE(3)$ stands for the rigid transformation from body-IMU to camera, known from caliberation
• $\oplus$ is the transformation operation of $SE(3)$ group over $\mathbb{R}^3$ elements

I am unable to build visual intuition of whats going on here from this description. After some pondering, I came up with following description for each term with corresponding frame conversions.

1. $T_{CB}$ defines how the camera is positioned and oriented with respect to the body or vehicle on which it is mounted. $$\text{Camera frame}\rightarrow \text{IMU body frame}$$

2. $T_i^{-1}$ is the inverse tranformation matrix for the camera pose at time $i$. $T_i$ is a camera's pose in camera frame transformed from world frame. $T_i^{−1}$ effectively undoes this transformation, effectively transforming camera pose from camera frame back to the world frame before projection function can project it to image frame. $$\text{Camera frame} \rightarrow \text{World frame}$$

3. $x_j$ represents a point in the 3D global or world frame. It is obtained in the $\text{World frame}$.

4. $\oplus$ combines two poses. Here both of its operands ($T_{CB}T^{-1}_i$ and $x_j$) are in the world frame. $$\text{World frame}\rightarrow \text{World frame}$$

5. $\prod$ is a projection function. It maps first maps 3D point $x_j$ from world frame to the camera frame and then projecting it onto image frame. $$\text{World frame}\rightarrow\text{Camera frame}\rightarrow\text{Image frame}$$

6. $u_{ij}$ is the observed image coordinates, typically measured in the camera frame. To compare them to the projected coordianates , they are tranformed to the image frame. $$\text{World frame}\rightarrow\text{Camera frame}\rightarrow\text{Image frame}$$

7. $r_{ij}$ is a reprojection error in image frame. The reprojection error represents the discrepancy between the observed image coordinates $u_{ij}$ and the projected image coordinates $\prod(…)$, both of which are expressed in the image frame.

Doubts

Q1. What exactly $T_{CB}T^{-1}_i$ does "intuitively / visually"? I thought it should be doing $$\text{Body frame}\rightarrow\text{Camera frame}\rightarrow\text{World frame}$$, so that it can be then combined with $x_j$ (which is in $\text{World frame}$) by $\oplus$. But $T_{CB}$ is $$\text{Camera frame}\rightarrow \text{IMU body frame}$$ and not $$\text{IMU Body frame}\rightarrow\text{Camera frame}$$. Well, I am confused here! Can someone help me with this?

Q2. To be honest, I really dont fully get what $\oplus$ does exactly "intuitively / visually". So, point 4 can be very well wrong above. Can someone please explain what it does "intuitively / visually"?

Answer 1

I agree with you it is not the best notation.

Regarding your understandings.

1. Is correct. It is Transform from the camera frame to the IMU/Body Frame.
2. $T_i$ does not involve the camera. You can look up near Equation 1 and see that it is the pose of the IMU/Body. In Visual inertial odometry the main frame we estimate is the IMU, not the camera. Changing the notation $T_{W,B}$ might be more appropriate. The transform from the World frame to the Body Frame.
3. Three is correct. Though to make my notation easier I am going to call it $x_{w}$. I am going to drop the $j$ since we are just working with 1 point for this equation.
4. I want to call this an abuse of notation. It is the operation that a transform applies to a 3D point so rotates it and translates it. It is equivalent to the following.

$$ T_u * x_a = R_u * x_a +t_u $$

Here $T_u$ is the transform broken up into its Rotational($R_u$) and translational components($t_u$). So simply rotates and then translates the point.

5. Mostly correct. A projection functions maps from the Camera frame to the image.
6. Yes, $u_{ij}$ is observed image coordinate. But since you are already observing them in the image you don't need to transform them into a new frame. They are already in the image frame.
7. Correct.

I am going to rewrite the equation using my notation.

Notation:

• $W$ is world frame
• $B$ is Body/IMU frame
• $C$ is Camera frame

$$ u_{ij} - \prod(T_{CB} * T_{WB}^{-1} * x_W )$$

Which utilizing the inverse operator(you just swap the subscripts) is equal to

$$ u_{ij} - \prod(T_{CB} * T_{BW} * x_W )$$

Notice how in this form all subscripts are adjacent to each other with the same letter: $T_{BW}$ is adjacent to $x_W$ for $W$, and $T_{CB}$ * $T_{BW}$ the $B$ is adjacent. This is how you can verify an equation is correct.

If you want a wordy explanation.

Given a point in world. Transform it into the Body frame, transform it into the Camera frame. Project it from the camera frame to the image frame. Subtract the estimated point from your observation.

Q2: I think you actually understand this. It is just a transformation applied to a point, but they used somewhat weird notation.