# Is there an algorithm using the Kinect depth image (not the point cloud) for registration?

I know given the intrinsics fx, fy, cx, cy (where fx, fy are the horizontal and vertical focal length, and (cx, cy) is the location of principal point of the camera if pinhole camera model assumed) of an Kinect depth camera(or other range sensor?), a depth pixel px=(u, v, d) ((u, v) is the pixel coordinate, d is the depth value) can be converted to a 3D point p:

p=(x, y, z) x=(u-cx)/fx*d y=(v-cy)/fy*d z=d  so that a depth image can be converted to a point cloud, and indeed, a depth Image represents a unique point cloud physically.

SLAM systems e.g. KinectFusion use such point clouds for ICP based registration to obtain camera pose at each time and then fuse new point cloud to the previously reconstructed model.

However, my mentor told me that depth Image cannot be inveribly converted to a point cloud since it's 2D->3D mapping with ambiguity (which I disagree), and he claims that I should use the depth Image at time (i-1) and (i) for registration, not the derived point cloud.

（If I have to obey my mentor's order) I've been reading papers and found one using Gradient Descent to solve camera pose (tx, ty, tz, qw, qx, qy, qz):

Prisacariu V A, Reid I D. PWP3D: Real-time segmentation and tracking of 3D objects[J]. International journal of computer vision, 2012, 98(3): 335-354.

which uses RGB Images and a known model for pose estimation. However, I've NEVER found a paper (e.g., KinectFusion and other later RGB-D SLAM algorithms) deals with depth data just as plane image but not point cloud for registration. So could someone give me some hint (papers or opensource code) about:

How to do depth image registration without converting them to point clouds?

• Can you clarify what the meaning is of the terms you're using? For instance, what are fx, fy, cx, cy, u, v, and d? Also, what exactly are you trying to accomplish? I would argue that the depth image represents a unique point cloud when paired with the corresponding camera model, because two identical depth maps with different lenses on the camera mean that two distinctly different scenes were recorded. Ultimately I think I would argue with you but there could be nuances to your particular application that I think you're glossing over that could decide one way or the other what's required. Jun 2, 2016 at 15:36
• @Chuck, sorry ;), edited. Shortern my question: to my knowledge, SLAM and Visual Odometry algorithms always use point clouds for frame registration and camera localization, is there an algorithm which uses raw depth image for registration? Jun 2, 2016 at 16:23

Consider this - what is the process of doing SLAM? First, get some sensor data, then move in the world, get some more sensor data, then try to do feature identification and matching to build up a map.

You could, theoretically, still build a map in "depth" units instead of in meters or feet or whatever units your camera model spits out. After all, what is a coordinate frame but some arbitrary choice?

However, all the algorithms you read about do use the camera models to convert to Cartesian "world" coordinates because that's a "common language" that is "spoken" by most other sensors.

To that point, if you are going to attempt SLAM, maybe a wheel encoder would help determine how much distance has been traveled. BUT, if you (or your advisor) is insisting on using "native" depth units instead of converting those back to world coordinates via the camera transform, then that means that you instead need to convert your wheel encoder into depth units.

That is, you can either convert maps in depth units to Cartesian units, or you can convert every other sensor from Cartesian to depth units. GPS, magnetometers, wheel encoders, etc.

You could, of course, do SLAM without any external input, but then what? Well, then you get your entire map of the world and the localization of the sensor in terms of depth units. So what does it mean to be 1.5 depth units from a wall? Can you do anything useful with that?

In conclusion I would argue that, if you are insisting on not converting to proper world coordinates, then you are doing SLAM for the sole purpose of a SLAM exercise and not because you intend on doing anything useful, because the resulting map and localization doesn't have any external meaning.

• thx for your apply! Is your key point "depth unit" lacking real world scale? but isn't kinect (or Xtion etc.)'s output depth map of millimeters unit? Sorry if I misunderstand u ;) Jun 5, 2016 at 14:37
• @zhangxaochen - The depth map may have engineering units, but that's only half of the puzzle. If I told you an object was 1 meter away at pixel (1,1), what would that mean? It depends on the mapping between the 2D pixel plane and the 3D space - i.e., the camera model. With a regular lens it might be in front of you, but with a fisheye lens it might be above you. If you knew that an object was 15 pixels wide in your depth map, how far does the wheel need to turn to pass it? Well, what is the FOV of the camera? Again, put the camera in world coordinates or put everything else in the camera's. Jun 7, 2016 at 12:35
• the camera model is given and fixed, I mean, with regular camera of certain intrinsics fx, fy, cx, cy and regardless of distortion params, How to use depth image as an intensity image directly for registration? My advisor told me the reason for this is that he'd like to treat depth and color the same way so that they could be written in one energy function, yet I don't think it's the right way that other known RGB-D SLAM algorithms use (maybe there is indeed a paper with such an idea but I didn't find it?) Jun 8, 2016 at 2:34