# Pictures coordinates to robot coordinates

I have a picture on my C# software and the corresponding object in real world on 2d plane of motorized XY axis table. What I want is when I select a pixel on my picture, the motorized table should be moving to the exact same point of the real world object. The real world object and its picture may have distortion and rotation differences and the picture on the software is not perfect too.

I use 4 points on each to map all the points in between, example:

Normal picture (x,y): [250, 25] , [250, 287] , [399, 287] , [400, 28] (in pixels)

Table coordinates (x,y): [0, 0] , [2098, 29538] , [19127, 28164] , [17097, -1200] (in microsteps)

I tried using OpenCV's homography:

I used FindHomography() to get the H matrix and I transform the picture point choosed using PerspectiveTransform() which give me the corresponding point in microsteps on the real world to move the motorized XY axis table.

OpenCvSharp.Mat hCv = OpenCvSharp.Cv2.FindHomography(srcPoints, dstPoints);
OpenCvSharp.Point2d[] resultCv = OpenCvSharp.Cv2.PerspectiveTransform(targetCv, hCv);


I also manualy calculated the homography matrix using this anwser : https://math.stackexchange.com/a/2619023 But I both cases I always get an error when transforming one of the four reference point like for [250, 25] the corresponding point should be [0, 0] but instead I get something like [-25, 245].

Question : I there a different way to link pictures coordinates to real world coordinates accuratly ?

Edit, more explanations:

To get my 8 points I select four points of the picture and then in real world I move my table to the four corresponding points manually. Let say I took a picture of my smartphone. I will get my points on the four edges of the phone. Then if I choose the X, Y pixel position corresponding to the front camera of my phone, my motorized table should move to get a landmark above the front camera of my real phone.

• What are the unit/thing you count and how ? – N. Staub May 29 '18 at 11:30
• Picture is in pixels and the coordinates of the motorized table unit is microsteps. – Dairon May 29 '18 at 11:36
• what type of coordinate are you using (I assume Cartesian) – N. Staub May 29 '18 at 11:36
• Only X,Y on both – Dairon May 29 '18 at 11:43

First note that for px =250 (picture x) you get two different table values tx=0 and tx=2098, then for a difference of 1 unit in px you get a difference of around 2000 in tx (table x). From that it seems that your measurement is not super reliable. I would suggest you to center the table reading on the center of the equivalent pixels. For py=287 you get two values separated by approx. 1500 units on the table.

If you know that the mapping is linear you should just find the coefficients such that $px = \alpha ~tx$ and $py = \beta ~ty$. To do so, consider $n_{px}$ the number of pixel along x-direction and $n_{tx}$ the number of motor counts from the center of the first pixel to the center of the last one. Then $\alpha = (n_{px} -1 ) / n_{tx}$. Doing that on the full length of the table should give you better accuracy than doing it for two adjacent pixels. Then you just need to determine the offset between the two set of coordinate (i.e. put one set at [0,0] an see the value of the other).

At that point you have everything to construct the affine relation between your set of coordinates. You might want to watch carefully your sign and offset based on what you define as origins on both the table and picture (if both are the same corner it is easier).

If you want you can write this affine relation in matrix form.

$\begin{bmatrix} px \\ py \end{bmatrix} = \begin{bmatrix} \alpha & 0 \\ 0 & \beta \end{bmatrix} \begin{bmatrix} tx \\ ty \end{bmatrix} + offset$

• I think you should edit your question with all the infos from comments and check the policy of SE about cross posting – N. Staub May 29 '18 at 12:13
• Then I would suggest you to make a summary of the other function of opencv suggested and which you tried – N. Staub May 29 '18 at 12:32
• Question edited, so I deleted the unnecessary comments. With my new explanations do you have another method to get the right coordinates on the table ? – Dairon May 29 '18 at 13:02
• I am not specialist of the method you tried in opencv but the fact that for the same pixel coordinate you get a difference of 1500/2000 motor counts is not going to help to retrieve the homography. Try to find points where for the same pixel value the motor count values are closer. – N. Staub May 30 '18 at 8:42