# Robotic vision eye-in-hand research

I started researching Robotic Vision topics for a while but I found myself a bit stuck. My background is in the Computer Vision and Machine Learning fields but I lack the knowledge in robotics.

My goal for now is to make an eye-in-hand system perform pick-and-place movements. Something like this: https://www.youtube.com/watch?v=GOY5VED54zQ

I'm pretty sure the key to achieve this is the calibration phase but out of all the resources I found I could not understand how this is done. I get that I need to find transformations between camera frame to robot frame but I have no idea how to get them. Any paper that I read about this gets me lost in the math part which I find very hard to understand for now.

What I know:

• how to deal with problems in the camera plane(i.e detect objects)
• how to control the robot which is an UR-3 cobot

What I don't know:

• how the camera world points relate to the robot world points

Is there a general approach to this problem? Is there any library that can help me with this?

Any study materials/tutorials or explanations are highly appreciated. Thank you in advance.

• Calibration is absolutely key. Generally,the process making an interpolation/correlation between robot joint coordinates and on screen pixels is not a simple process, hence the complicated maths, as the systems learns what each joint pulse is for a particular point in a given image, a interpolation function can be derived to determine pixel -> mm coordinates. A proper calibration is critical to determine an accurate positioning. Jan 6, 2020 at 19:49

What you are confused about are frame transformations.

If you have ever poked around with ROS, you will see that robot joint locations are determined through frame transformations.

These transformations are calculated because the current joint angles are known.

Those angles are then passed into what are called homogeneous transformation matrices. The matrices are essentially 4x4 matrices that multiply nicely together to keep track of the end-link orientation (upper left 3x3 part) and the tranlation aka. XYZ of the robot tip (upper right 3x1 vector in homog. matrix).

These matrices, before being given current joint angles, are often created from the Denavit-Hartenberg (D-H) parameters.

Since you are confused with all the math, and robotics is really just applied math, I suggest you start learning how D-H params work.

Then, once you understand how to get the tool-tip orientation and position from the joints, you can then link the camera frame with the base-frame of the robot you are working with.

Essentially, the camera relates to the rest of the world through a root frame, often called a world frame. All data points are synced to that frame.

Then, when you add XYZ points in the camera frame, you can automatically know where they are in Tool-tip frame, aka. End of Arm Tooling (EoAT) frame.

Then, if you mount a camera on the EoAT, you finally have the vision-in-eye setup you envision in your question title.

Does this make sense?

Note:

You asked about libraries? There are many matrix libraries out there, and saying which is better or worse is outside the scope of Robotics Beta and this question. There are Python libraries and C++ libraries: you will have to choose one. I know that there are many matrix libraries to be used with OpenGL, so I recommend looking at what computer graphics people like to use. An example might be GLM.

I should say though, that ROS uses quaternions, but those are faster to compute. They achieve the same thing as 4x4 homog. trans. matrices, with less CPU time.