I am working on a robotics project for fun and can't really wrap my head around a solution to my perception problem.

Setup: My robot will have a stereo vision setup and will have to detect certain objects and align itself to those objects in a certain pose. The robot will know what the width and height of those objects are. The robot will be using a tx1 for computation, so implementation needs to be pretty fast. Also, the environment's lightning will change a lot so using color for detection isn't a great option.

My plan: To use convolutional neural networks to detect those objects of interest. I have been able to program a network to detect those objects in 2D however, I am stuck on how to detect the pose of the objects in 3D. My idea has been to detect the object with the neural network and once I have that region of interest, get the point cloud. Then fit a 3D model using ICP. Once the 3D model is fit I can get the pose of the object.

I have also seen people using 3D correspondence grouping for this but would that work on a non dense point cloud that stereo vision generates?

I am pretty novice in this area and would love to get advice on from some more advanced robotics practitioners.

Thanks for you time!

  • $\begingroup$ i am not sure what you are asking $\endgroup$
    – holmeski
    Oct 11, 2016 at 3:20
  • $\begingroup$ your plan seems ok. I do not have experience with correspondence grouping. $\endgroup$
    – 50k4
    Oct 11, 2016 at 5:58
  • $\begingroup$ Do you have a prior pointcloud of all your objects? If so, ICP seems like a reasonable thing to do. $\endgroup$
    – abarry
    Nov 24, 2016 at 17:32

1 Answer 1


There are a few things that will make ICP hard to work with here:

  • ICP is very sensitive to the initialization it is given and is very likely to converge to a poor solution if it is given a bad starting point. Your point cloud will probably not give you a good initial guess for the orientation of the object.
  • It can be thrown off by points in your point cloud that are not part of the object. There are versions of ICP that try to be more robust to this, but it's hard to do well. i.e., your CNN detections will probably just provide a bounding box detection of the object, but the object will not entirely fill that box. The point cloud you extract will therefore contain points that are not part of the object, which could pull the translation estimate from ICP pretty far off too. You will probably need to do some kind of 3D segmentation to try to isolate points on the object, which will be tricky to tune to work in all the cases you encounter.
  • It can also be pretty expensive...

I'd recommend starting with something like RANSAC on interest points on the object. You can compute a set of distinctive points (e.g., corner detections or other SIFT-like features) on your object model and on your image within the bounding box provided by your CNN's 2D detections. The success of this approach will depend largely on whether your objects have good, distinctive features to detect. Then you can estimate 3D locations for each of these points from disparities in the neighborhood. Applying RANSAC to these 3D points should give pretty accurate and robust 3D pose estimates. After that, you could try using ICP to refine them if needed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.