Hand eye calibration of robot arm using a 2D camera and marker of known size

Question

If my understanding is correct, one flavor of hand eye calibration requires the following pairs of data for different robot arm positions:

1. The transform from the robot base to the robot hand.
2. The transform from the marker to the robot camera.

By transform, I mean the 4x4 homogeneous matrix that contains both rotation and translation information.

The part I am stuck on is getting the transform from the marker to the robot camera, which is a 2D camera. The marker can be checkerboard, or Aruco, etc. for which opencv can give you the pose of the marker with respect to the camera.

The question I have is fundamental: Is it even possible to get the pose of the camera from images of a marker of known size? Won't there be a scale ambiguity specifically in the z direction of the camera? Is this z ambiguity resolved by using a marker of known size? Finally, if this scale ambiguity is always there, is it possible to do hand eye calibration using a 2D camera and a marker of known size?

Answer 1

Is it even possible to get the pose of the camera from images of a marker of known size?

-> Yes. From the camera image that is containing a calibration board, you can estimate a 3 by 3 homography matrix H. Then, if you know a camera intrinsic parameters then you can inversely recover the scale known transformation matrix using the intrinsic and homography H. Here is a good reference for this (have a look at 3.1 Closed-form solution). Translation scale will be unknown when the board size is unknown.

Won't there be a scale ambiguity specifically in the z direction of the camera?

-> Not at all if the camera intrinsic and board size is known.

Is this z ambiguity resolved by using a marker of known size?

-> Yes

Finally, if this scale ambiguity is always there, is it possible to do hand eye calibration using a 2D camera and a marker of known size?

-> As your statement "scale ambiguity is always there" is not necessary, the calibration is fairly simple now with the scale known transformations above. You just need to find a robot base to the board transformation and hand to eye transformation.

You have 12 unknowns and each constraint will give you 3 equations. 4 different set data will be enough for a closed-form solution. What you need to solve is

$${^RT_{E_i}}{^ET_{C}}{^{C_i}T_{B}}={^RT_{B}} $$

where R is robot base, E is end-effector, C is a camera, B is the board coordinate.

Refers to the following paper for a detailed solution.

Answer 2

I'm tied up at the moment, so I'll flesh this more out tomorrow morning, but I believe the answer to your question is that the pose estimation algorithms utilize the camera parameters of the camera you're using. Field of view, resolution, focal point, and a known object's dimensions all come together to form similar triangles. The focal point and resolution form the triangle between the CCD or image receptor and lens of the camera, then there's another triangle between the lens and the actual object.

If you know the focal distance and "height" on the image receptor, then you can get the angular width of the object. Knowing its real height and the angular width then lets you calculate the distance or depth of the object.