The Benefit of Hand-Eye calibration with 2D camera

Question

I have a general question about the benefit of Hand-Eye calibration. I know that the Hand-Eye calibration can be done with many softawres by solving the AX=XB or AX=ZB equation. But my question is, how can I use this calibration information afterwards.

Say, I have an eye-to-hand calibration configuration where a 2D camera is stationary and a gripper is mounted on the robot arm. After doing the calibration I would have the trasformation from 2D Camera to the robot base. Now I would like to detect some object using the 2D Camera and send its position to the robot to grip it. But since I don't have the depth information for this object I wouldn't be able to calculate the 3d Position of the object in the robot base, even though I have the calibration information.

So for me it seems that using a single 2D camera isn't helpful to work with a robot arm. One should use a 3D camera to be able to extract useful information and then send them to the robot. Or am I missing something?

Answer 1

There are cases where the object's dimension or shape is known. Especially 2D camera and manipulator combination is widely used in manufacturing automation where you have the 3D cad model of the object at a high chance. If you can register the 3D cad information to its projection in the image you can calculate the distance + orientation of the object from your camera. This registration process is often called template matching.

Answer 2

If you use PNP (Perspective N Point), you could estimate the distance in Z if dimensions of the objects are known.