Mapping between pointcloud and image

Question

Hi everyone, For the past few days I have been working to use pointcloud information and the corresponding RGB image of the scene so that I can get a 1 to 1 mapping between objects present in both. I really could not figure it out. I had this idea(below), please share ur views about modifying or completely changing it.

The aim is to mark objects in both (pointcloud and corresponding RGB image of scene) with a box of same color for same objects.

First I thought of extracting the shape features of objects from the edges. For similar objects the shape features are going to remain more or less the same(based on orientation). Edge detection is possible only in a 2D image which I obviously did for the RGB image.

How to do edge detection in a pointcloud which is 3D representation of the scene? In a 3D pointcloud viewed in the same direction as the image is facing will definitely have objects whose shape or boundary look similar. Yes if you look at pointcloud at a different angle than the view of image ,the objects may appear to be of different shapes( eg, bus moving forward infront of you in RGB image(2D) will look squarer but when viewed sideways in pointcloud(3D) looks rectangular)

So I thought of converting the 3D pointcloud into a 2D image in the same viewing direction of RGB image(so that edges could be found). Is it really possible to represent the 3D pointcloud as a 2D image essentially depicting the surfaces and objects present. If yes, how to do it? All these pictures of pointcloud which I see in the Internet are like 3D objects with various shapes in a 2D plane. Are these 2D or a snapshot from the 3D viewer of various softwares??? Eagerly awaiting for ur ideas and views about this problem.

I went through the documentation in Tutorials/CloudToImage for converting pointcloud to image.I used the node "convert_pointcloud_to_image" But when i tried it i get an error :-

Error in converting cloud to image message: No rgb field!!

In the above are they trying to represent 3D world in a 2D image(that is what i need) or are they just trying change its type to that of sensor_msgs/Image??

I dont know if my approach is correct but please pour in your thoughts about doing this pointcloud to objects in image mapping in some other way if its better or simpler than my approach. Will be truly grateful to you.

Thank you,

Karthik

---

Originally posted by KARTHIK MURUGAN on ROS Answers with karma: 48 on 2012-12-10

Post score: 0

Answer 1

what is the source of your data? if you have the camera parameters i.e K matrix or M matrix you could simply find the 2d point(image point) by

p = K[R|T] * P

P being your 3d point and and R and T are your camera rotation and translation.see camera calibration for finding out your camera parameters and the viewpoint is usually stored in the pointcloud header.

if the camera is not available and you have a 3d point for every image point( highly unlikely unless it is a synthetic dataset) you could find the 3d to 2d relation using the image edges and corresponding 3d points (given the viewpoint you can find out the extreme end points in your 3d data, join the 3d point and camera center those with maximum angle should correspond to image end points). then use svd or any other method( min 6 points) (known to have some error).

also you could try to add random colors to your points (using pcl::pointXYZRGB) and then convert it to image, once you have the image match it to the original image and find the original colors(doubt that'll work but you can give it a try)

good luck

---

Originally posted by sharky with karma: 36 on 2014-02-28

This answer was ACCEPTED on the original site

Post score: 0