# Why do strong rotations affect Monocular Vision based Visual Odometry?

In context of the paper Semi Dense Visual Odometry for a Monocular Camera J Engel, J Sturms, D Cremers

At page 7, last paragraph the author writes

The achieved tracking accuracy for two feasible sequences - that is, sequences which do not contain strong rotations without simultaneous translation - is given in Table 1.

Here the Author implies that the method is not feasible (fails ?) on datasets that contain strong rotation. In the paper, Visual Odometry is performed in two parts inverse depth estimation of points and tracking of camera. I want to analyze how these parts are affected when the camera undergoes strong rotations?

Here is what I think (kindly correct me and or add missing details)

Part I
Inverse depth estimation is based on disparity among two successive frames. The observed pixel disparity is equal to scaled inverse depth for the pixel.

When camera undergoes strong rotation it causes apparent pixel disparity however the pixel would still be at the same depth wrt the camera. So this results in false disparity and subsequently wrong depth estimation (?)

Part II
The tracking of camera is being done via image alignment (using depth map). The image alignment is formulated as a photometric error minimization problem:

$$E(\xi) := \sum_{i}\frac{\alpha(r_i(\xi))}{\sigma_{d_i}^2}r_i(\xi)$$ where $$r_i(\xi) = (I_2(~{\color{blue}w(x_i, d_i, \xi)}~) - I_1(x_i))^2$$ and $${\color{blue}w}$$ is a warp function that maps each point $$x_i$$ in reference image $$I_1$$ to the respective point in the new image $$I_2$$

The warp function requires depth $$d_i$$ to calculate the new position of a pixel in image $$I_2$$ that was observed earlier in $$I_1$$. With wrong depth estimate the minimization problem would yield wrong value for $$\xi$$ (?)

Also, are there techniques to deal with pure camera rotation in case of Monocular Vision based Odometry?

• The depth perception with a monocular image in the way the authors did it can only work if there is more than one image of the same scene, from slightly different viewpoints (as would be the case for binocular vision - the two simultaneous images of binocular vision are replaced by two concurrent images at slight offset of camera position, off course betting on the mostly unchanging scene between two or more successive pictures). A pure rotation is not delivering this changed view point of the scene, and can thus not be used for depth perception – bukwyrm Aug 5 at 8:20
• @bukwyrm Thanks for the answer. Intuitively, what you say is the basis for temporal stereo with monocular camera. As you can see in Part I the disparity (which also results from the offset of camera position) is still there in the observed scene in presence of a rotation. In that case the cause for the failure of the method is not apparent (disparity due to rotation should produce wrong depth estimate). And, in Part II, it appears to me that the image alignment problem should work fine given only rotations, but would it really work? I am trying to get a deeper insight. – vvy Aug 6 at 2:31
• Sorry, i thought you did not get the technique, when in truth i did not get your explication.. Still, i don't get what you miss: The author says they can't work with strong rotations that do not at the same time contain translations - that is pure, strong rotations. In case of pure rotations, any depth map that is produced has no basis in reality, because it is exclusively made up from rotation-induced errors. There simply is no temporal stereo with no translation. And without a good, frame-filling depthmap, your rot&trans estimation on the next frame will be worthless. – bukwyrm Aug 6 at 5:27