# How to perform Image alignment using pyramid levels

To gain some confidence, I want to implement the camera tracking (optimization problem) discussed in Semi Dense Visual Odometry for a monocular cameraJ Engel, J Strum, D Cremers

$$E(\xi) = \underset{i}\Sigma\frac{\alpha(r_i(\xi))}{\sigma_{d_i}^2}(r_i(\xi))$$ $$r_i(\xi) = (I_2(w(x_i, d_i, \xi)) - I_1(x_i))$$

Using Gauss-Newton method, as discussed in the same paper, (local optima) $$\xi$$ between two monocular images can be found. Here is how I think the process goes:

• Start with initial guess $$\xi_0$$ (~ 0), Images $$I_1, I_2$$, Depth for image $$I_1$$ (for now I am using a RGB-D dataset)

• Start at lowest level (most coarse) $$L_n$$ pyramid image (same as shrinked image?) and corresponding depth map (for points in shrinked image?)

• Gauss Newton iterations with some library (I plan to use python for ease).

• Setup the residual calculation function that takes input $$I_2$$, $$I_1$$, {$$x_i$$}, {$$d_i$$} and {$$\xi$$} and produces $$r_i$$. The Jacobian involved will be calculated numerically.
• I can skip $$\alpha(.)$$ and $$\sigma_{d_i}^2$$ for now. $$E(\xi) = \underset{i}\Sigma ~r_i^2$$
• At the end I expect to get the result $$\xi_L$$
• Use the solution $$\xi_L$$ as initial guess and repeat for the next pyramid level (finer level)

Questions:

• Am I missing some step in above process ? Please let me know.

• Are there tools that do this ?

• In general how it is done ?

PS: can someone with higher reputation add the tag "Image-alignment" to this question?