Pictures coordinates to robot coordinates

Question

I have a picture on my C# software and the corresponding object in real world on 2d plane of motorized XY axis table. What I want is when I select a pixel on my picture, the motorized table should be moving to the exact same point of the real world object. The real world object and its picture may have distortion and rotation differences and the picture on the software is not perfect too.

I use 4 points on each to map all the points in between, example:

Normal picture (x,y): [250, 25] , [250, 287] , [399, 287] , [400, 28] (in pixels)

Table coordinates (x,y): [0, 0] , [2098, 29538] , [19127, 28164] , [17097, -1200] (in microsteps)

I tried using OpenCV's homography:

I used FindHomography() to get the H matrix and I transform the picture point choosed using PerspectiveTransform() which give me the corresponding point in microsteps on the real world to move the motorized XY axis table.

OpenCvSharp.Mat hCv = OpenCvSharp.Cv2.FindHomography(srcPoints, dstPoints);
OpenCvSharp.Point2d[] resultCv = OpenCvSharp.Cv2.PerspectiveTransform(targetCv, hCv);

I also manualy calculated the homography matrix using this anwser : https://math.stackexchange.com/a/2619023 But I both cases I always get an error when transforming one of the four reference point like for [250, 25] the corresponding point should be [0, 0] but instead I get something like [-25, 245].

Question : I there a different way to link pictures coordinates to real world coordinates accuratly ?

Edit, more explanations:

To get my 8 points I select four points of the picture and then in real world I move my table to the four corresponding points manually. Let say I took a picture of my smartphone. I will get my points on the four edges of the phone. Then if I choose the X, Y pixel position corresponding to the front camera of my phone, my motorized table should move to get a landmark above the front camera of my real phone.

Answer 1

First note that for px =250 (picture x) you get two different table values tx=0 and tx=2098, then for a difference of 1 unit in px you get a difference of around 2000 in tx (table x). From that it seems that your measurement is not super reliable. I would suggest you to center the table reading on the center of the equivalent pixels. For py=287 you get two values separated by approx. 1500 units on the table.

If you know that the mapping is linear you should just find the coefficients such that $px = \alpha ~tx$ and $py = \beta ~ty$. To do so, consider $n_{px}$ the number of pixel along x-direction and $n_{tx}$ the number of motor counts from the center of the first pixel to the center of the last one. Then $\alpha = (n_{px} -1 ) / n_{tx}$. Doing that on the full length of the table should give you better accuracy than doing it for two adjacent pixels. Then you just need to determine the offset between the two set of coordinate (i.e. put one set at [0,0] an see the value of the other).

At that point you have everything to construct the affine relation between your set of coordinates. You might want to watch carefully your sign and offset based on what you define as origins on both the table and picture (if both are the same corner it is easier).

If you want you can write this affine relation in matrix form.

$\begin{bmatrix} px \\ py \end{bmatrix} = \begin{bmatrix} \alpha & 0 \\ 0 & \beta \end{bmatrix} \begin{bmatrix} tx \\ ty \end{bmatrix} + offset $