Is there an algorithm using the Kinect depth image (not the point cloud) for registration?

Question

I know given the intrinsics fx, fy, cx, cy (where fx, fy are the horizontal and vertical focal length, and (cx, cy) is the location of principal point of the camera if pinhole camera model assumed) of an Kinect depth camera(or other range sensor?), a depth pixel px=(u, v, d) ((u, v) is the pixel coordinate, d is the depth value) can be converted to a 3D point p:

p=(x, y, z) x=(u-cx)/fx*d y=(v-cy)/fy*d z=d so that a depth image can be converted to a point cloud, and indeed, a depth Image represents a unique point cloud physically.

SLAM systems e.g. KinectFusion use such point clouds for ICP based registration to obtain camera pose at each time and then fuse new point cloud to the previously reconstructed model.

However, my mentor told me that depth Image cannot be inveribly converted to a point cloud since it's 2D->3D mapping with ambiguity (which I disagree), and he claims that I should use the depth Image at time (i-1) and (i) for registration, not the derived point cloud.

（If I have to obey my mentor's order) I've been reading papers and found one using Gradient Descent to solve camera pose (tx, ty, tz, qw, qx, qy, qz):

Prisacariu V A, Reid I D. PWP3D: Real-time segmentation and tracking of 3D objects[J]. International journal of computer vision, 2012, 98(3): 335-354.

which uses RGB Images and a known model for pose estimation. However, I've NEVER found a paper (e.g., KinectFusion and other later RGB-D SLAM algorithms) deals with depth data just as plane image but not point cloud for registration. So could someone give me some hint (papers or opensource code) about:

How to do depth image registration without converting them to point clouds?

Answer 1

Consider this - what is the process of doing SLAM? First, get some sensor data, then move in the world, get some more sensor data, then try to do feature identification and matching to build up a map.

You could, theoretically, still build a map in "depth" units instead of in meters or feet or whatever units your camera model spits out. After all, what is a coordinate frame but some arbitrary choice?

However, all the algorithms you read about do use the camera models to convert to Cartesian "world" coordinates because that's a "common language" that is "spoken" by most other sensors.

To that point, if you are going to attempt SLAM, maybe a wheel encoder would help determine how much distance has been traveled. BUT, if you (or your advisor) is insisting on using "native" depth units instead of converting those back to world coordinates via the camera transform, then that means that you instead need to convert your wheel encoder into depth units.

That is, you can either convert maps in depth units to Cartesian units, or you can convert every other sensor from Cartesian to depth units. GPS, magnetometers, wheel encoders, etc.

You could, of course, do SLAM without any external input, but then what? Well, then you get your entire map of the world and the localization of the sensor in terms of depth units. So what does it mean to be 1.5 depth units from a wall? Can you do anything useful with that?

In conclusion I would argue that, if you are insisting on not converting to proper world coordinates, then you are doing SLAM for the sole purpose of a SLAM exercise and not because you intend on doing anything useful, because the resulting map and localization doesn't have any external meaning.