# How is the depth estimation done in DTAM / LSD-SLAM (direct VSLAM)?

I know that DTAM and LSD-SLAM both employ the photometric error between two keyframes to estimate the image depth. But I don't know what the photometric error describes precisely. In the DTAM paper, I found the following figure describing the principle:

However, if I only have images I_r and I_m and not the pose translation, how is the cost volume estimated? And how do I know in which row in I_m I have to look for my original Pixel I_r(u,v)?

• DTAM is an RGBD SLAM so it gets its depth from the depth camera. LSD-SLAM is a monocular slam method so it only calculates depth up to scale. So you need to be more clear what you want. Jun 27, 2020 at 16:17
• According to the paper, DTAM is a monocular system: "As a single hand-held RGB camera flies over a static scene, we estimate detailed textured depth maps at selected keyframes to produce a surface patchwork with millions of vertices" and "This model is composed of depth maps built from bundles of frames by dense and sub-pixel accurate multi-view stereo reconstruction" that the depth map is generated. I want to know how the direct depth estimation works for monocular images. Jun 27, 2020 at 17:03
• Ahh sorry got it confused with KinectFusion which is Richard Newcombe's other dense odometry work. Jun 27, 2020 at 20:19

I can only describe the general idea. I can not tell you exactly how they solve it mathematically.

if I only have images I_r and I_m and not the pose translation, how is the cost volume estimated?

You know that the cost volume lies in a space between depth e_max, e_min, and width u and height v. Now you take every small "cube" in this cuboid of size (e_max-e_min) x u x v and project it into your two images l_m and l_r. Then you have one pixel in image A (pixelA) and one pixel in image B (pixelB).

Then you calculate the photometric error. I am not sure how they do it here. You could go and sum the red, green and blue values of pixelA and sum the red,green and blue values of pixelB. Then you subtract these values. If the difference is small then the color or intensity might be similar. This might tell you that the cube is at the correct place.

And then you do this for every single of these little cubes. And calculate the average.

Now you say "what? but how do I know how to project the little cube to the second image?". The first image is easy. That is just 3d-backpropagation. But for the second image, you need the pose transformation. And here a minimization procedure comes into place.

Your transformation can be described by 6+ variables (depending on whether you use Euler angles, rotation matrices etc.). You want to estimate these variables. If there was a perfect solution then you could do this using gauss jordan elimination.

You have some initial guess of your transformation. This is of course not a good guess. Therefore the photometric error described above is really high. If you guess multiple times then you might see that the error increases or decreases.

This procedure is described in the paper in detail. It does the same as gauss jordan but for many variables. There is no perfect solution so gauss jordan can not work here and it would be far too slow.

Maybe this helps.

• Thanks! I guess my biggest problem was understanding that you only have an estimate in the beginning, which might be random or based off a previous estimate and is continuously improved. Maybe the image lead me into thinking the solution was already present somehow. Aug 10, 2020 at 15:30