# Depth camera pixel values to 3D coordinates

I have a robotic arm 6-DOF and an Intel RealSense depth camera. I want to go from pixel values to 3D world coordinates, is it possible with eye-in-hand calibration? If yes can you guide me on how to perform this calibration? I have done some study on my own but have a lot of doubts.

• What do you mean by pixel values? Commented Mar 26, 2021 at 8:31

The Realsense provides you with the depth $$d$$ corresponding to the pixel $$(u,v)$$. Knowing from the forward kinematics of the robot arm the $$4 \times 4$$ homogeneous transformation $$\mathbf{T}$$ from the root reference frame to the 3D frame of the camera attached to the end-effector, the 3D coordinates of the point $$\mathbf{p}$$ projected in the pixel at hand are:
$$\mathbf{p} = \mathbf{T} \cdot \left( \frac{d}{f} \cdot \left( u-\frac{w-1}{2} \right), \; \frac{d}{f} \cdot \left( v-\frac{h-1}{2} \right), \; d, \; 1 \right)',$$
where the resolution of the camera is $$w \times h$$ pixels, while $$f$$ is the camera focal length that you can retrieve from the Realsense API.
Clearly, $$\mathbf{T}$$ depends on the current configuration of the arm joints.