I'm pretty sure this is the right place for this question, but if not, please point me in the right direction!

Basically, I would like to know how do SLAM using multiple cameras and sensors that are asynchronous. Along with this, I'm curious if there's any way to use the SLAM information—or any way in general—to automatically align the pose data from multiple devices into the same coordinate system.

My specific situation is that I have devices that each have 2 cameras, 2 IMUs without built-in magnetometers, a magnetometer, and a GPS. The cameras use fisheye lenses, point different directions (but have a little overlap on their images), and capture images at the same time. However, all other sensors take measurements at different times. The IMUs have similar capture rates but don't capture at the same time, the magnetometer takes measurements much less often that the IMUs do, etc. I have the transformation matrices to convert between the sensors' local coordinate systems, along with other intrinsic and extrinsic data.

I'd like to get the orientations and positions from all of this data as well as a mapping of the environment. I also have recordings of the devices in the same location, so I want to put the devices' poses in the same reference frame so that they can be visualized at the same time. The algorithm doesn't have to do online / real-time SLAM, since I'm only analyzing the data after it's all been captured. Furthermore, it's not necessary for the algorithm to use all of the sensors, but I would like it to at least use both cameras. If possible, I'd also prefer the algorithm to not rely on ROS.

Most of the algorithms I found online didn't match my case exactly, and the one that I think might've doesn't have a public implementation that I can use.

  • $\begingroup$ Have you looked at rtabmap? $\endgroup$ Nov 29, 2022 at 22:48

2 Answers 2


I have done many multi-modal sensor fusions with LiDAR, IMU, and cameras. Obviously, synchronization is the hardest problem if it is not synchronized at the HW level.

To be able to do SLAM with multi-modal asynchronous sensors, you need a few things in your SLAM system.

  1. Continuous time trajectory representation

1.1 If your trajectory can be represented as a function of time, you can represent your sensor estimation w.r.t the system time. And you need to have cost functions that take sensor data + sensor time.

E.g the poupular P2P ICP error is represented as e = Tp-p' where your continuous time representation of this equation is simply e = T(t)p-p'.

Look at my papers if you want to know the details. I have done it for LiDAR, camera, and IMU in the paper but it works for Stereo-Inertial problems too.  



1.2 An alternative way is integrating IMU and assuming that the cameras are synchronized by the trigger signal. This mitigates small differences but you still need to measure the time difference between IMU and cameras. The most popular reference for this method would be OpenVINS.

  1. Time-lag estimation

Now you have a tool that represents the trajectory in continuous time to deal with the asynchronous estimation. But you also have to find the time lag between the sensors. Thanks to the continuous-time representation you just have to add the estimated time lag dt to each sensor's estimation.

Just add the time lag to your system state. Now your cost function will look like  e = T(t+dt)p-p'

  1. Enough motion that can observe the time lag between the sensorsNote that estimating the time lag between the sensors won't be possible unless you move the sensor. The time lag is observable only when you move the sensor

I’m making assumptions from the question … I guess the system has two phones fixed with duct tape ;)

So, I would calibrate sensors to the camera and calibrate camera to camera.

I would have visual odometry per camera and fuse with each sensor (with Kalman filter).

Get 3D with SFM and use bundle adjustment to minimize the errors in the camera to camera pose.

It sounds easy but this is a PhD thesis.

https://publikationen.bibliothek.kit.edu/1000080527/20396421[3D Visual Perception for Self-Driving Cars using a Multi-Camera System: Calibration, Mapping, Localization, and Obstacle Detection]


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.