4
$\begingroup$

In Visual Inertial Odometry, assuming the camera and the IMU are attached to the same rigid body, why isn't it enough to know just the relative rotation between the camera and the IMU? Why do we need to know the relative translation too?

Imagine a situation where we know the exact relative rotation ${^I_C}R$ from camera's frame of reference to IMU's frame of reference, but we don't know the relative translation ${^I_C}t$ between them. Where in a VIO pipeline would the relative translation ${^I_C}t$ be used and why VIO cannot work without it?

Isn't the point of camera-IMU extrinsics to transform the angular velocity and acceleration measurements from the IMU frame of reference to the camera frame of reference? Velocity and acceleration aren't points, but vectors, so adding an arbitrary offset ${^I_C}t$ to them doesn't make sense to me.

To simplify the situation, let's assume ${^I_C}R$ to be identity so the axes of camera's and IMU's frame of references are aligned, but assume there is a shift ${^I_C}t$ between the camera and the IMU.

My understanding so far is that the rotational component of the motion experienced by the IMU and the camera is the same, regardless of the pivot point of the rotation. I can only imagine ${^I_C}t$ being of use when somehow correlating gyro and accelerometer readings to identify the pivot point of the rotation experienced by the IMU.

By knowing the location of the pivot point of the rotation relative to the IMU and knowing camera-IMU extrinsics, it would be possible to get a better inter-frame translation estimate, but separating acceleration caused by rotation from other sources of acceleration seems difficult to me.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

You answered yourself. Your underlying model of point moving through space usually assumes the center of gravity to match the camera frame. If your inertial sensor have a translational offset from the camera attached to a rotating rigid body, you would measure extra tangential and radiant acceleration components that your camera should not experience.

You can remove them by using the gyroscope angular rates $\omega$ and knowing the offset $r$. This should be the relationship between the accelerations measured in the imu frame and camera frame:

$$a^{\text{cam}} = a^{\text{imu}} - \dot{\omega} \times r - \omega \times \omega \times r$$

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for your answer. I'm a bit confused. Is the point of knowing the camera-IMU offset in a VIO pipeline just to correct the acceleration vector so that we can compute the acceleration that's experienced in camera's frame of reference? Is there no other use for the camera-IMU offset? So if a hypothetical VIO pipeline wanted to use only gyroscope data, then knowing only the relative rotation between camera and IMU would be enough and knowing the relative offset would be useless? $\endgroup$
    – jordi
    Jan 31 at 18:04
  • $\begingroup$ I have follow-up questions, if you don't mind: Is $\omega \times \omega \times r$ the extra centripetal acceleration and $\dot\omega \times r$ the extra tangential acceleration? What exactly is the $r$ vector? Is it the offset from the camera to the IMU? $\endgroup$
    – jordi
    Jan 31 at 18:07
  • $\begingroup$ Correct! Try to match your camera model with your inertial model for better vio $\endgroup$
    – SystemSigma_
    Feb 1 at 7:32
  • $\begingroup$ @jordi the formula above assumes no relative (and constant) rotation between camera and imu frames (i.e. $\omega = \omega^{\text{cam}} = \omega^{\text{imu}}$). If this is not the case, use a standard rotation matrix to first correct the gyro+accl vectors (a simple change of base) and then compensate accelerations in the camera frame with such formula using rotated gyro data $\omega^{\text{cam}}$ $\endgroup$
    – SystemSigma_
    Feb 1 at 10:55
  • 1
    $\begingroup$ You are correct, i guess. Though you should consider to place the IMU near the center of mass (i.e. pivot rotating point) of your robot/drone dynamic model (falling into case 1). If this is not the case, you should correct again (with the same formula) IMU readings to the true center of mass point with a proper offset $r^{\text{imu}}$. Ideally, you want all measurements to be measured in with the robot's center of mass. so that you can model properly with differential equations the dynamics $\endgroup$
    – SystemSigma_
    Feb 1 at 16:18
1
$\begingroup$

It's important to know the offset between the camera and the IMU because the IMU's information is only useful when it can be correleated with the camera data.

In a lot of the trivial cases this difference could be negligible. However depending on your requirements for accuracy you will need to compensate.

Lets use a thought experiment of a baseball batter. And simplify it down to a 1m bat mounted on a motor. With the IMU mounted at the pivot point (aka by the hands) and the camera at the tip of the bat. And you want to be tracking the incoming baseball visually. And for simplicity lets put the ball on a tee just out of reach of the bat so it's not moving.

When the camera observes the ball it will be experiencing translational movement as well as rotational movement. The ball will appear to be approaching the camera on an arc. But by the construction of the experiment we know that the IMU is only reporting rotational motion. Such that if you ignore the offset between the camera and the IMU at any give point as you will miss estimate the velocity of the ball. This is most obvious when the bat is passing very close to the ball. The camera will believe that the ball is moving at approximately $v = \omega \times r$ because the only estimated motion of the camera is the rotation $w$ and I'm assuming the ball in negligably far away from the camera $ \epsilon $, implying that $\epsilon \times \omega \sim 0$ Thus the velocity of the stationary ball would be computed to be approximately the velocity of the camera because the camera's motion is not being properly compensated for. And if you're trying to track this ball or anything else in the scene, integrating these incorrect velocities over time will cause your whole understanding of the environment to diverge.

Now this is showing the worst case of the ratios of $r$ and $\epsilon$ but this is the math that is required to maintain high accuracy. In some cases you can make arguments that ignoring these components won't be a problem. But unless you've fully made that case and know that it will never be invalid in the future, it's generally considered best to do the full computation as it can be done relatively quickly in closed form and you don't have to put caveats into your system about the ratio of the distance from the IMU to the camera to the objects in the view.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your answer. I'm not sure if I understood your answer well, but it seems like in your example, it is assumed that the IMU is always in the center of rotation (in the pivot point). In this special case, it does make sense to me why knowing the camera-IMU offset is beneficial (to estimate translation in camera's frame of reference). $\endgroup$
    – jordi
    Jan 31 at 18:13
  • $\begingroup$ I used that case of only rotation around the IMU because that's the case where the errors are most obvious to explain. There would be similar errors of the camera was at the base and the IMU was at the tip. And any other offset configuration if you don't use a very limited set of motions that remain in the null set. (Off the top of my head I would guess that would be only pure translation.) I'm showing that the model without that term is inadequate using the example of this simple case. $\endgroup$
    – Tully ♦
    Feb 2 at 3:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.