How to combine ORB-SLAM3 with robot_localization

Question

I'm using robot_localization package to estimate the pose of my robot wrt odom frame. I managed to fuse both data coming from my wheel odometry and IMU.

My robot has a fisheye camera and I have used ORB-SLAM3 to generate a map of the environment (although I have scaling issues as expected) and get an estimate of the camera poses with respect to an inertial reference frame.

Both work quite well, separately. I'm using ROS1.

From robot_localization package I get the transformation between odom and base_link frames. From ORB-SLAM3 I get the transformation between map and camera frames.

I would like to combine the two together to get a better estimate of the robot pose.

However, I'm not quite sure how to do that.

I have been searching online but I could not find any example of using ORB-SLAM3 in combination with robot_localization.

Also, I'm not quite sure that this could be possible.

In essence, I would like to understand how to use the two to estimate odom-->base_link and map-->odom transformations and what it is missing in the pipeline to obtain those transformations.

Any help would be really appreciated. I'm also happy to uploade yaml config files of both my ekf (implemented by robot_localization) and my visual monocular slam (implemented using ORB-SLAM3).

Thank you, Timothy

Answer 1

In order to fuse the orb SLAM pose estimate with the Kalman Filter of the robot_localization package it is necessary to publish PoseWithCovariance messages.

An implementation in orb2_slam is available here. Did anyone make a version for orb2_slam?

Another solution is to use orb3_slam in a monocular-inertial setting. In such a case, orb3_slam will also publish odometry_msgs.

In common.cc file

if (sensor_type == ORB_SLAM3::System::IMU_MONOCULAR || sensor_type == ORB_SLAM3::System::IMU_STEREO || sensor_type == ORB_SLAM3::System::IMU_RGBD)
{
    odom_pub = node_handler.advertise<nav_msgs::Odometry>(node_name + "/body_odom", 1);
}

I'll re-calibrate my monocular camera with the imu and then run the mono_inertial version of orb3_slam.