Accessing rviz view (Camera + Marker) in rospy

Question

I want to get the rviz view, which is showing me visualization marker on the top of camera view in a single image. Please see below the rviz view:

I tried to view the image topic but it doesn't contain any marker image (as expected). Please see below the image:

Later, I echo the rostopic but couldn't get any clue to get the image. Below is the output of rostopic:

$ rostopic echo /visualization_marker
header: 
  seq: 401
  stamp: 
    secs: 149
    nsecs: 462000000
  frame_id: kinect2_rgb_optical_frame
ns: basic_shapes
id: 8
type: 1
action: 0
pose: 
  position: 
    x: -0.151297542
    y: 0.000741795717294
    z: 0.44793458834
  orientation: 
    x: 0.95382124573
    y: -0.00821160004013
    z: 0.241985041143
    w: 0.177766252933
scale: 
  x: 0.045
  y: 0.045
  z: 0.009
color: 
  r: 0.5
  g: 0.0
  b: 0.5
  a: 1.0
lifetime: 
  secs: 1
  nsecs:         0
frame_locked: False
points: []
colors: []
text: ''
mesh_resource: ''
mesh_use_embedded_materials: False

I want to get the image (in the form of sensor_msgs/Image) as it is seen in rviz using python in ros.

---

Originally posted by ravijoshi on ROS Answers with karma: 1744 on 2017-03-30

Post score: 0

---

Original comments

Comment by lucasw on 2017-03-30:
Does the Marker appear in rviz in the regular 3D view? If so make an rviz Axes and set it to the camera frame_id, and make sure the z-axis is pointed in the general direction of the marker you want to see overlaid on the Image.

Comment by ravijoshi on 2017-03-30:
If I disable (unselect checkbox) camera, I can see the visualization marker. However it appears bigger and misplaced in the screen. But once I enable camera view, I can see marker in proper place (on the top of camera image)

Comment by huanxiner on 2017-03-30:
I am not sure if I understand you completely, but are you looking for a topic with 'rendered' camera+marker images? If so I don't believe it is readily available in rviz..

Comment by ravijoshi on 2017-03-30:
@huanxiner: Please see the first image in the question. This image is being displayed by rviz. Notice that the identified marker has been colored in this picture by rviz. I want to get this image (including the colored/detected marker) in ros python code.

Answer 1

If I understand your problem correctly then rviz_camera_stream https://github.com/lucasw/rviz_camera_stream/ may do part of what you want- it can publish an rviz view of the marker from the same perspective as your camera. What it doesn't do currently is underlay the video. That wouldn't be too hard to add (by borrowing code from the rviz Camera plugin).

Instead of adding that feature you could have a python node subscribe to both the source video (undistorted/rectified) and an rviz_camera_stream where you have set the background color to black and are only visualizing the Marker, and combine the two in the node (create a mask with the non-background pixels in the rviz image and then copy the marker pixels to the source camera image).

---

Originally posted by lucasw with karma: 8729 on 2017-03-30

This answer was ACCEPTED on the original site

Post score: 4

---

Original comments

Comment by ravijoshi on 2017-03-31:
I can easily subscribe and get the camera image (video) which I am able to see using rosrun image_view image_view image:=/kinect2/hd/image_color. However, I am not able to get any marker image. rostopic echo /visualization_marker shows me pose in 3D space. How to convert it to camera space?

Comment by lucasw on 2017-03-31:
Are you running rviz_camera_stream?

Comment by ravijoshi on 2017-04-01:
What do you mean by saying running rviz_camera_stream? I posted two pictures in the question above. First one is rviz view and another one is the output of following command rosrun image_view image_view image:=/kinect2/hd/image_color. Can you please explain in detail?

Comment by lucasw on 2017-04-03:
something like this:

cd ~/catkin_ws/src
git clone https://github.com/lucasw/rviz_camera_stream.git

Then build it and try out:

roslaunch rviz_camera_stream demo.launch

There will be images published out of rviz with this.