To gain some confidence, I want to implement the camera tracking (optimization problem) discussed in Semi Dense Visual Odometry for a monocular cameraJ Engel, J Strum, D Cremers

$$E(\xi) = \underset{i}\Sigma\frac{\alpha(r_i(\xi))}{\sigma_{d_i}^2}(r_i(\xi))$$ $$r_i(\xi) = (I_2(w(x_i, d_i, \xi)) - I_1(x_i))$$

Using Gauss-Newton method, as discussed in the same paper, (local optima) $\xi$ between two monocular images can be found. Here is how I think the process goes:

  • Start with initial guess $\xi_0$ (~ 0), Images $I_1, I_2$, Depth for image $I_1$ (for now I am using a RGB-D dataset)

  • Start at lowest level (most coarse) $L_n$ pyramid image (same as shrinked image?) and corresponding depth map (for points in shrinked image?)

  • Gauss Newton iterations with some library (I plan to use python for ease).

    • Setup the residual calculation function that takes input $I_2$, $I_1$, {$x_i$}, {$d_i$} and {$\xi$} and produces $r_i$. The Jacobian involved will be calculated numerically.
    • I can skip $\alpha(.)$ and $\sigma_{d_i}^2$ for now. $E(\xi) = \underset{i}\Sigma ~r_i^2$
    • At the end I expect to get the result $\xi_L$
  • Use the solution $\xi_L$ as initial guess and repeat for the next pyramid level (finer level)


  • Am I missing some step in above process ? Please let me know.

  • Are there tools that do this ?

  • In general how it is done ?

PS: can someone with higher reputation add the tag "Image-alignment" to this question?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.