I've recently been trying to use Gazebo to do some modelling for a couple tasks. I have a robot that's effectively able to locate a ball and get x,y coordinates in terms of pixels using a simple RGB camera from the Kinect. I also have a point cloud generated from the same Kinect, where I hope to find the depth perception of the ball using the X,Y coords sent from the circle recognition from my RGB camera. My plan earlier was to convert the X,Y coordinates from the RGB camera into meters using the DPI of the Kinect, but I can't find any info on it. It's much, much harder to do object recognition using a Point Cloud, so I'm hoping I can stick to using an RGB camera to do the recognition considering it's just a simple Hough Transform. Does anybody have any pointers for me?
-
$\begingroup$ The approach I posted earlier doesn't work, I wasn't thinking. $\endgroup$– IcheCommented Nov 7, 2015 at 2:11
2 Answers
The critical part is the registration between depth data and RGB data. If the registration is calibrated properly then you can just extract the depth for the particular target pixel (X,Y), using interpolation for sub-pixel coordinates. See this answer for help with the registration -- it is a common problem that has already been solved.
Once you have the depth for the pixel coordinates of interest, you can easily get the Cartesian coordinates relative to the camera using the camera matrix (based on the field of view and image size). Note that you also have to account for distortion in the RGB camera. This OpenCV documentation page should help start you off, and there are plenty of resources out there addressing transformation between camera pixel coordinates, camera homogeneous coordinates, and actual world Cartesian coordinates given the pixel depth.
-
$\begingroup$ A great, straight to the point response! This is exactly what I was looking for! Thank you. $\endgroup$– IcheCommented Nov 7, 2015 at 2:47
-
$\begingroup$ Excellent, hopefully those resources are enough and you can connect all the dots. Are you using the Kinect v1 or v2? $\endgroup$ Commented Nov 7, 2015 at 2:55
-
$\begingroup$ I'm using an Openni Kinect. I'm just doing simulations in Gazebo at the moment but I'll probably end up using a v2 Kinect when I get all the simulations done. $\endgroup$– IcheCommented Nov 7, 2015 at 3:05
-
$\begingroup$ If you do end up using the Kinect v2 then you might be able to use the infrared intensity image to locate the target -- in which case you can avoid needing to register between depth and RGB (since the depth data is produced from the same infrared camera data). The problem of converting from pixel coordinates to world coordinates then shifts to the infrared camera configuration instead of the RGB configuration. $\endgroup$ Commented Nov 7, 2015 at 3:17
Very common problem which calls for using the "Pointcloud Library" PCL: www.pointclouds.org This datastructures in the library also make it easy to find according 3D-Points when you segmented your RGB image. There are lots of tutorials and a very busy community.
