I've recently been trying to use Gazebo to do some modelling for a couple tasks. I have a robot that's effectively able to locate a ball and get x,y coordinates in terms of pixels using a simple RGB camera from the Kinect. I also have a point cloud generated from the same Kinect, where I hope to find the depth perception of the ball using the X,Y coords sent from the circle recognition from my RGB camera. My plan earlier was to convert the X,Y coordinates from the RGB camera into meters using the DPI of the Kinect, but I can't find any info on it. It's much, much harder to do object recognition using a Point Cloud, so I'm hoping I can stick to using an RGB camera to do the recognition considering it's just a simple Hough Transform. Does anybody have any pointers for me?
The critical part is the registration between depth data and RGB data. If the registration is calibrated properly then you can just extract the depth for the particular target pixel (X,Y), using interpolation for sub-pixel coordinates. See this answer for help with the registration -- it is a common problem that has already been solved.
Once you have the depth for the pixel coordinates of interest, you can easily get the Cartesian coordinates relative to the camera using the camera matrix (based on the field of view and image size). Note that you also have to account for distortion in the RGB camera. This OpenCV documentation page should help start you off, and there are plenty of resources out there addressing transformation between camera pixel coordinates, camera homogeneous coordinates, and actual world Cartesian coordinates given the pixel depth.
Very common problem which calls for using the "Pointcloud Library" PCL: www.pointclouds.org This datastructures in the library also make it easy to find according 3D-Points when you segmented your RGB image. There are lots of tutorials and a very busy community.