# Depth camera pixel values to 3D coordinates

I have a robotic arm 6-DOF and an Intel RealSense depth camera. I want to go from pixel values to 3D world coordinates, is it possible with eye-in-hand calibration? If yes can you guide me on how to perform this calibration? I have done some study on my own but have a lot of doubts.

• What do you mean by pixel values? – Ralff Mar 26 at 8:31

## 1 Answer

The steps are pretty easy to implement!

The Realsense provides you with the depth $$d$$ corresponding to the pixel $$(u,v)$$. Knowing from the forward kinematics of the robot arm the $$4 \times 4$$ homogeneous transformation $$\mathbf{T}$$ from the root reference frame to the 3D frame of the camera attached to the end-effector, the 3D coordinates of the point $$\mathbf{p}$$ projected in the pixel at hand are:

$$\mathbf{p} = \mathbf{T} \cdot \left( \frac{d}{f} \cdot \left( u-\frac{w-1}{2} \right), \; \frac{d}{f} \cdot \left( v-\frac{h-1}{2} \right), \; d, \; 1 \right)',$$

where the resolution of the camera is $$w \times h$$ pixels, while $$f$$ is the camera focal length that you can retrieve from the Realsense API.

Clearly, $$\mathbf{T}$$ depends on the current configuration of the arm joints.

Bear in mind that the standard reference frame of the camera has the z-axis orthogonal to the image plane and pointing outward, the x-axis pointing rightward, and the y-axis pointing downward as specified by the pinhole camera model.