In Visual Inertial Odometry, assuming the camera and the IMU are attached to the same rigid body, why isn't it enough to know just the relative rotation between the camera and the IMU? Why do we need to know the relative translation too?
Imagine a situation where we know the exact relative rotation ${^I_C}R$ from camera's frame of reference to IMU's frame of reference, but we don't know the relative translation ${^I_C}t$ between them. Where in a VIO pipeline would the relative translation ${^I_C}t$ be used and why VIO cannot work without it?
Isn't the point of camera-IMU extrinsics to transform the angular velocity and acceleration measurements from the IMU frame of reference to the camera frame of reference? Velocity and acceleration aren't points, but vectors, so adding an arbitrary offset ${^I_C}t$ to them doesn't make sense to me.
To simplify the situation, let's assume ${^I_C}R$ to be identity so the axes of camera's and IMU's frame of references are aligned, but assume there is a shift ${^I_C}t$ between the camera and the IMU.
My understanding so far is that the rotational component of the motion experienced by the IMU and the camera is the same, regardless of the pivot point of the rotation. I can only imagine ${^I_C}t$ being of use when somehow correlating gyro and accelerometer readings to identify the pivot point of the rotation experienced by the IMU.
By knowing the location of the pivot point of the rotation relative to the IMU and knowing camera-IMU extrinsics, it would be possible to get a better inter-frame translation estimate, but separating acceleration caused by rotation from other sources of acceleration seems difficult to me.