Hi everyone.
I do want to register my depth image onto an mochrome image using depth_image_proc register.
I do publish the transform between the frame_ids using a static tranform publisher (see launch file).
This all seems to work fine in general as my registered depth topic is valid in rviz and receives frames. However, they are only black. Thereby, my depth image has a resolution of 848 x 480, whereas my monochrome image only has 640 x 512 px.
Also, to proof it works, I registered the depth image to another rgb camera (with a fitting resolution) and it worked there.
The rectified images look fine to me, so I think the internal calibrations should be good.
My guess is, that this is either a problem with the external calibration (that I did with an external tool due to the different resolutions) or could the resolutions itself be a problem for the remapping?
Do you maybe have hints how to debug this, as I do not really know how to proceed anyomore?
Thanks in advance.
my launch file:
<launch>
<arg name="namespace_ir" default="gas_camera"/>
<arg name="namespace_vis" default="realsense"/>
<!-- gas_cam -->
<include file="$(find gas_cam)/launch/gas_cam.launch">
<arg name="namespace" value="$(arg namespace_ir)" />
</include>
<!-- realsense -->
<include file="$(find realsense2_camera)/launch/rs_camera.launch" ns="$(arg namespace_vis)">
<arg name="filters" value="pointcloud" />
<arg name="tf_prefix" value="rs" />
</include>
<!-- tf2 -->
<node pkg="tf2_ros" type="static_transform_publisher" name="rs_2_ir" args="-0.062249 -0.083060 0.006821 0.00465763 0.052355 0.02046292 rs_link ir_kam_frame" />
<!-- stereo proc for image registration -->
<node pkg="nodelet" type="nodelet" name="nodelet_manager" args="manager" />
<node pkg="nodelet" type="nodelet" name="nodelet1" args="load depth_image_proc/register nodelet_manager">
<remap from="depth/camera_info" to="/$(arg namespace_vis)/camera/depth/camera_info"/>
<remap from="rgb/camera_info" to="/$(arg namespace_ir)/camera_info"/>
<remap from="depth/image_rect" to="/$(arg namespace_vis)/camera/depth/image_rect_raw"/>
</node>
<!-- RVIZ -->
<node pkg="rviz" type="rviz" name="rviz" args="-d $(find gas_cam)/rviz/default_visualization.rviz"/>
</launch>
Edit (due to @lucasw 's comments): Here the camera infos. Please don't care about the different time stamps - the synchronization works (I just copied them using rosecho).
One thing to know when comparing the different camera infos is that the camera differ stongly in their angle of fiew:
- ir_kam f ~ 25 mm (12° halfangle)
- realsense f~ 2mm (I'd guess 50° halfangle)
camera info depth cam:
header:
seq: 67
stamp:
secs: 1640873968
nsecs: 393739462
frame_id: "rs_depth_optical_frame"
height: 480
width: 848
distortion_model: "plumb_bob"
D: [0.0, 0.0, 0.0, 0.0, 0.0]
K: [422.76092529296875, 0.0, 422.03045654296875, 0.0, 422.76092529296875, 238.31170654296875, 0.0, 0.0, 1.0]
R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
P: [422.76092529296875, 0.0, 422.03045654296875, 0.0, 0.0, 422.76092529296875, 238.31170654296875, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi:
x_offset: 0
y_offset: 0
height: 0
width: 0
do_rectify: False
camera info (not working) monochrome cam:
header:
seq: 6
stamp:
secs: 1640874318
nsecs: 647664546
frame_id: "ir_kam_frame"
height: 512
width: 640
distortion_model: "plumb_bob"
D: [-0.8905452676, 2.0011537585, -0.0077582393, -0.0107622419, -25.5474403138]
K: [1649.1023644655, 0.0, 361.7890567933, 0.0, 1648.4837741689, 261.1807393191, 0.0, 0.0, 1.0]
R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
P: [1649.1023644655, 0.0, 361.7890567933, 0.0, 0.0, 1648.4837741689, 261.1807393191, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi:
x_offset: 0
y_offset: 0
height: 0
width: 0
do_rectify: False
camera info rgb camera
header:
seq: 31
stamp:
secs: 1640874456
nsecs: 314450502
frame_id: "rs_color_optical_frame"
height: 720
width: 1280
distortion_model: "plumb_bob"
D: [0.0, 0.0, 0.0, 0.0, 0.0]
K: [922.0919799804688, 0.0, 652.3343505859375, 0.0, 922.7329711914062, 358.4428405761719, 0.0, 0.0, 1.0]
R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
P: [922.0919799804688, 0.0, 652.3343505859375, 0.0, 0.0, 922.7329711914062, 358.4428405761719, 0.0, 0.0, 0.0, 1.0, 0.0]
binning_x: 0
binning_y: 0
roi:
x_offset: 0
y_offset: 0
height: 0
width: 0
do_rectify: False
Rviz setup of monochrome camera: (/image_rect topic):
Edit2: A screenshot of rviz, showing the cameras of the rectified depth map, the rectified monochrome camera and the registered depth map (that is black ;-) (please don't wonder about the different objects in the image - one is live and the other loads simulated images).
Originally posted by max_ros on ROS Answers with karma: 36 on 2021-12-29
Post score: 0
Original comments
Comment by lucasw on 2021-12-29:
It should be fine to go to a different resolution (though you may want to use fill_upsampling_holes, but you'll at least want to see some pixels are getting set and not just pure black), but there could be something wrong with the camera_info coming out of the monochrome image- can you echo a single camera_info from the not-working monochrome camera and the working rgb camera and add those to your question? (You could add a depth camera camera_info also)
If you set up an rviz Camera configured to use the monochrome camera_info does rviz show a reasonable view through it?
Probably isn't your problem but I've always had strange issues with tf2_ros static_transform_publisher and have always gone back to the older tf1 version rather than debug it to the point of asking a good question here or opening an issue or even fixing it.
Comment by max_ros on 2021-12-30:
Hi @lucasw, thanks for the quick answer and your thoughts!
I edited the question to provied the info you asked for. As the shown frames look fairly reasonable on the grid, I also don't think tf2_static_publisher is the problem...
Comment by lucasw on 2021-12-30:
Try setting the monochrome camera distortion to all zeros like the other cameras- that final distortion coefficient of -25 doesn't seem right.
Another test of the monochrome camera calibration is to put the monochrome camera through the image rectification nodelet to see if the undistorted output looks reasonable.
The monochrome camera fx/fy parameters of 1649 (where the other cameras look to have much wider angles-of-view) makes me wonder if that camera is pointed at small part of the depth image where there are no depth points. Or are there depth values covering the whole image?
Comment by max_ros on 2021-12-30:
Monochrome camera calibration:
I guess you're right about the -25, this seems strange. I did another calibration, that has 0.xxx value in this position. However, the screenshot that I added to the question shows the rectified monochrome image still with this 25 number. Setting the distortion coefficients all to 0 doesn't change the seen behaviour of the black registered depth map.
dense depth map: good idea! However, I think the screenshot shows an example of an dense depthmap - and the registered depth map stays black...
Comment by lucasw on 2021-12-30:
Another debug is to use https://github.com/lucasw/vimjay/blob/master/scripts/dr_camera_info.py (or otherwise manually publish a fictional camera info) instead of your real camera_info and make the numbers similar to the working camera or the not-working camera and anywhere in-between see if anything there causes the problem.
Additionally you could hook up an interactive tf publisher (https://github.com/lucasw/rviz_interactive_tf/blob/master/scripts/ddr_tf.py and there are other options) instead of using the static transform publisher and see if moving frames around causes the problem, the trivial case is where there is no difference at all from depth to other camera, then slowly moving it further away or rotate.
Comment by max_ros on 2022-01-03:
First: These dynamic reconfigure tools are awesome, thanks a lot for sharing!
Second: happy new year^^
Third: Interesting observation: Before playing around with the interactive tf publisher I though to check the trivial case. And even there, using my calibration data, I get a black images only for the registered depth image. When using your dynamic camera_info script, I manage to finally see registered depth data, so I guess it must be the calibraiton :-)
P.S. I haven't yet set the fill_upsampling_hole parameter. As I understand from this , this is as parameter which isn't yet documented in the wiki? If this is correct (sorry for the newby question) - how would I set a nodelet parameter in the launch file?
