Object Detection and Pose Estimaion

Question

I am working on a robotics project for fun and can't really wrap my head around a solution to my perception problem.

Setup: My robot will have a stereo vision setup and will have to detect certain objects and align itself to those objects in a certain pose. The robot will know what the width and height of those objects are. The robot will be using a tx1 for computation, so implementation needs to be pretty fast. Also, the environment's lightning will change a lot so using color for detection isn't a great option.

My plan: To use convolutional neural networks to detect those objects of interest. I have been able to program a network to detect those objects in 2D however, I am stuck on how to detect the pose of the objects in 3D. My idea has been to detect the object with the neural network and once I have that region of interest, get the point cloud. Then fit a 3D model using ICP. Once the 3D model is fit I can get the pose of the object.

I have also seen people using 3D correspondence grouping for this but would that work on a non dense point cloud that stereo vision generates?

I am pretty novice in this area and would love to get advice on from some more advanced robotics practitioners.

Thanks for you time!

Answer 1

There are a few things that will make ICP hard to work with here:

• ICP is very sensitive to the initialization it is given and is very likely to converge to a poor solution if it is given a bad starting point. Your point cloud will probably not give you a good initial guess for the orientation of the object.
• It can be thrown off by points in your point cloud that are not part of the object. There are versions of ICP that try to be more robust to this, but it's hard to do well. i.e., your CNN detections will probably just provide a bounding box detection of the object, but the object will not entirely fill that box. The point cloud you extract will therefore contain points that are not part of the object, which could pull the translation estimate from ICP pretty far off too. You will probably need to do some kind of 3D segmentation to try to isolate points on the object, which will be tricky to tune to work in all the cases you encounter.
• It can also be pretty expensive...

I'd recommend starting with something like RANSAC on interest points on the object. You can compute a set of distinctive points (e.g., corner detections or other SIFT-like features) on your object model and on your image within the bounding box provided by your CNN's 2D detections. The success of this approach will depend largely on whether your objects have good, distinctive features to detect. Then you can estimate 3D locations for each of these points from disparities in the neighborhood. Applying RANSAC to these 3D points should give pretty accurate and robust 3D pose estimates. After that, you could try using ICP to refine them if needed.