I know given the intrinsics
fx, fy, cx, cy (where
fx, fy are the horizontal and vertical focal length, and
(cx, cy) is the location of principal point of the camera if pinhole camera model assumed) of an Kinect depth camera(or other range sensor?), a depth pixel
px=(u, v, d) (
(u, v) is the pixel coordinate,
d is the depth value) can be converted to a 3D point
p=(x, y, z)
so that a depth image can be converted to a point cloud, and indeed, a depth Image represents a unique point cloud physically.
SLAM systems e.g. KinectFusion use such point clouds for ICP based registration to obtain camera pose at each time and then fuse new point cloud to the previously reconstructed model.
However, my mentor told me that depth Image cannot be inveribly converted to a point cloud since it's 2D->3D mapping with ambiguity (which I disagree), and he claims that I should use the depth Image at time (i-1) and (i) for registration, not the derived point cloud.
（If I have to obey my mentor's order) I've been reading papers and found one using Gradient Descent to solve camera pose
(tx, ty, tz, qw, qx, qy, qz):
Prisacariu V A, Reid I D. PWP3D: Real-time segmentation and tracking of 3D objects[J]. International journal of computer vision, 2012, 98(3): 335-354.
which uses RGB Images and a known model for pose estimation. However, I've NEVER found a paper (e.g., KinectFusion and other later RGB-D SLAM algorithms) deals with depth data just as plane image but not point cloud for registration. So could someone give me some hint (papers or opensource code) about:
How to do depth image registration without converting them to point clouds?
fx, fy, cx, cy, u, v, and d? Also, what exactly are you trying to accomplish? I would argue that the depth image represents a unique point cloud when paired with the corresponding camera model, because two identical depth maps with different lenses on the camera mean that two distinctly different scenes were recorded. Ultimately I think I would argue with you but there could be nuances to your particular application that I think you're glossing over that could decide one way or the other what's required. $\endgroup$