To gain some confidence, I want to implement the camera tracking (optimization problem) discussed in Semi Dense Visual Odometry for a monocular cameraJ Engel, J Strum, D Cremers
$$E(\xi) = \underset{i}\Sigma\frac{\alpha(r_i(\xi))}{\sigma_{d_i}^2}(r_i(\xi))$$ $$r_i(\xi) = (I_2(w(x_i, d_i, \xi)) - I_1(x_i))$$
Using Gauss-Newton method, as discussed in the same paper, (local optima) $\xi$ between two monocular images can be found. Here is how I think the process goes:
Given
- Start with initial guess $\xi_0$ (~ 0), Images $I_1, I_2$, Depth for some points in image $I_1$ is given
Steps
Start at lowest level (most coarse) $L_n$ pyramid image (same as shrinked image?) and corresponding depth map (for points in shrinked image?)
Gauss Newton iterations with some library (I plan to use python for ease).
- Setup the residual calculation function that takes input $I_2$, $I_1$, {$x_i$}, {$d_i$} and {$\xi$} and produces $r_i$. The Jacobian involved will be calculated numerically.
- I can skip $\alpha(.)$ and $\sigma_{d_i}^2$ for now. $E(\xi) = \underset{i}\Sigma ~r_i^2$
- At the end I expect to get the result $\xi_L$
Use the solution $\xi_L$ as initial guess and repeat for the next pyramid level (n-1)
Questions:
Am I missing some step in above process ? Please let me know.
Is there a library function in openCV that take an image (and its depth image) as input and give in output the requested pyramid level (for a choosen n) image as output (depth image will also need to be shrinked)?
PS: can someone with higher reputation add the tag "Image-alignment" to this question?