Rviz: Display image in 3D scene

Question

Hi

For a visual odometry project I'm using Rviz to display what my filter is doing. I.e. I'm displaying the current camera pose, the path the camera has taken and the features it is tracking as a point cloud. I would also like to show the current camera image in the scene, attached to the cameras pose as it moves through around.

So far I have not been able to accomplish this. I can show the image in an Image display in Rviz, which is just a separate view/window, similar to what I would get with cv::imshow(). I also played around with creating a point cloud of all my pixels and assigning grayscale intensity values to it. This technically works, but it does not look very nice and is very slow, as is to be expected.

Is there a way to do what I'm trying to accomplish?

Any help is much appreciated

---

Originally posted by NickDP on ROS Answers with karma: 148 on 2015-10-01

Post score: 3

Answer 1

For DRC we developed a rviz Display type that takes in a Image message and shape_msgs/Mesh. It then texture maps the image on the mesh using the OGRE texturing API. So if you publish a flat rectangle mesh to it and simultaneously select the appropriate image topic, you should basically get what you're looking for. Unfortunately, the Mesh message type doesn't provide a frame_id and the plugin code currently lives in a pretty big repo, so I'd recommend copying the plugin code and implementing your own slightly modified version of the plugin.

Code here.

---

Originally posted by Stefan Kohlbrecher with karma: 24361 on 2015-10-07

This answer was ACCEPTED on the original site

Post score: 2

---

Original comments

Comment by NickDP on 2015-10-12:
That looks promising. Thank you very much

Comment by lucasw on 2016-02-08:
I'm trying that customized mesh display plugin route: if I have a square mesh in the xy plane, a projector set to [0,0,1] and [ 0. , 0.70682518, 0. , 0.70738827] should I see the texture on the plane? Currently I can see my mesh but no texture. And the frame_id of the image doesn't matter?

Comment by Stefan Kohlbrecher on 2016-02-08:
IIRC the mesh is always diplayed as if it's frame_id was the currently selected frame_id in rviz. The frame_id of the image is used. Our use case was transforming the mesh into world frame and then projecting image data based on the tf chain from camera to world.

Comment by natsu on 2017-07-23:
Hi, I'm trying to do dynamic texturing on a mesh(actually a 3D model in .dae format) does this plugin add textures only to a plane mesh or does it work for any arbitary mesh??

Comment by Stefan Kohlbrecher on 2017-08-09:
Works for arbitrary meshes.

Answer 2

https://github.com/lucasw/rviz_textured_quads could do what you want though you would have to create a tf child of the camera pose at the desired distance to the camera tf. The texture would be provided via an image topic. The core code is in fact a couple steps removed from that DRC plugin @stefan-kohlbrecher links to, but I wanted a simple image topic on only a rectangle interface, maybe the general Mesh projection could be re-introduced later or be in a different plugin.

Also answered in a similar question here: http://answers.ros.org/question/227304/dynamic-texture-in-rviz/

jsk_rviz_plugins CameraInfo has an option to display an image from the proper frame_id, at an adjustable fixed distance from the camera origin, and properly scales it with distance (an option to optionally intersect the camera info with chosen other frame instead of the fixed distance would be super useful, maybe I can add that).

---

Originally posted by lucasw with karma: 8729 on 2016-10-27

This answer was NOT ACCEPTED on the original site

Post score: 1

---