0
$\begingroup$

I have camera feed (in the form of RGB images) from 3 cameras with partially overlapping Field-of-view i.e. for the time stamp 0 to 100, I have total 300 frames or say synchronized 100 RGB frames for each camera.

The object (Robot) is moving from one place to another place. I don't know about the camera locations but for each frame & for each camera, I have 8-corner points of 3D bounding-box which are just 2D projections of corresponding 3D camera points on an image. For example, in frames, depending upon the time stamp, if the object appears then I have 2D coordinates of the 8 corners of the blue (3D) bounding box. Note that, in the below image, I have shown only an example object not the entire frame!

enter image description here

image source

Apart from this, I have an Intrinsic Matrix which is same for all the cameras. I also know the object (Robot) dimensions (length, breadth, & height).

So, using these RGB frames from all the cameras, 2D coordinates of 8 corner-points and object (Robot) dimensions, how can I calibrate 3 cameras as well as how can I find the poses of each camera w.r.t. the first camera in the form of a 4x4 transformation matrix [SE(3)]?

Edit 1:
After the calibration, how can I get the global 3-DOF pose of the robot (center of 3D bounding box) for all the 100 frames? For this purpose, I'm assuming the pose of the first camera as a global coordinate frame.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The process you need to go through is actually similar to the camera calibration procedure in OpenCV or other software. The chessboard is replaced by your robot, and you can skip the intrinsic estimation step. I would actually recommend you take a look https://github.com/hengli/camodocal a multi rig camera calibrator.

Anyways a high level overview.

The two steps you have to take are:

  1. Initial Pose Estimation
  2. Refinement via Bundle Adjustment.

Step 1:

You actually only need 1 frame for this. Probably take the synced images that has the most the most projected points in common. (Minimum is 3 points, but you really need each camera to see the 4 same points).

  1. Define your reference frame/origin in your robot/object. You will be estimating the camera positions relative to this. You now also have the 3D bounding box corner positions relative to this frame. If you define it to be the center of the object then a point might look like $[\frac{width}{2},\frac{length}{2},\frac{height}{2}]$

  2. Take your pick of PnP algorithm, and estimate the camera poses(Se3) individually. The 3D points are your bounding box corners relative to your origin. The projections are the 2D coordinates in your image. If you pick the origin to be the center of your robot then you now have calculated the pose of your camera with respect to the center of the robot.

  3. Do some matrix multiplication to convert the camera poses in the object frame, to be relative to the first camera pose.

$$ T_{1,2}=T_{o,1}^{-1}*T_{o,2} $$

Should look like that. Here $o$ is your object coordinate frame, and $1,2$ refer to camera 1 and camera 2.

If your cameras only partially overlap(e.g. only cameras 1,2 and 2,3 have overlaps) then do the same steps for each pair, and then just chain the transforms.

$$T_{13} = T_{1,2}*T_{2,3} $$

Step 2:

I will say this step might be optional for you. You already have the camera positions from step 1, so this just helps refine the results.

Essentially you just need to set up a large Bundle Adjustment problem and solve it using something like Ceres.

  1. Build your 3D Pointcloud. This pointcloud is composed of your bounding box corners at every timestamp. So in total you should have maximum $8*100=800$ points(probably less because sometimes a point isn't visible).

How to exactly do this is tricky. If your robot has perfect odometry then you can just multiply your points by the odometry transform. You can run an object pose estimator algorithm for all timestamps in camera 1. You can use the PnP algorithm again. You just need all 800 3D corner positions in a common reference frame, and there are different ways of doing that.

  1. Build your optimization problem in something like Ceres. Your cost function terms should link the 3D points and the cameras that observe it. See the camodocal code for examples of this.

  2. Solve the bundle adjustment problem.

Improve this answer
$\endgroup$
9
  • $\begingroup$ thank you thank you so much. Also, sorry for the late reply but I have been trying to implement what you suggested. I hadn't thought that I can use the robot dimension to get the 3d coordinates of 8 points & then use PnP algo. I think, as of now, step 1 would be enough. However, I have a few confusions: (1) If I keep my reference frame at robot center then it would be moving, right? and in my case, cameras are only partially overlapping. So, while calculating transformation between cam 1 & 2 and that of between 2 & 3, the robot would be at different place! Would that be OK? $\endgroup$
    – Raj
    Aug 30, 2020 at 6:38
  • 1
    $\begingroup$ Correct the robot/reference frame is moving. This doesn't matter for step 1 because you are only using 1 image at a time to calculate the delta transform. For step 1 you only need 2 images. Image 1 with overlap between cam 1&2 to calculate $T_{1,2}$, and Image 2 with overlap cam 2&3 to calculate $T_{2,3}$. $\endgroup$
    – edwinem
    Aug 30, 2020 at 15:37
  • 1
    $\begingroup$ Question 2 Answer. There are different ways to do this. The 2 possible methods I can think of are to do PnP again, or triangulation. For the PnP method you just run PnP at every timestep which lets you calculate $T_{oc}$. If the camera viewing the robot is camera 1 then your object pose is just simply the inverse $T_{o,c=1}^{-1}$. If it is camera 2,3 then you also do the inverse but also have to multiply it by the position of the camera in frame 1. Something like $T_{1,2}*T_{o,c=2}^-1$ $\endgroup$
    – edwinem
    Aug 30, 2020 at 15:46
  • 1
    $\begingroup$ For triangulation you need to be able to view and associate the bounding box points across 2 cameras. You the triangulate the points 3D position via something like the DLT algorithm. And then figure out the object center given the bounding box 3D points. $\endgroup$
    – edwinem
    Aug 30, 2020 at 15:50
  • 1
    $\begingroup$ Finally there is a whole field of algorithms called object pose estimation which you could research that specifically deal with this problem. $\endgroup$
    – edwinem
    Aug 30, 2020 at 15:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.