0
$\begingroup$

I am just starting a research project at my university involving sensor fusion between a Camera and a LiDAR device to detect and localize cars around our vehicle (bounded boxes around cars in 3D point-cloud). We need a rig having one Camera and one LiDAR to capture data, which we will label manually, and later we will train a deep neural model that takes the advantage of color intensity and depth information for increased accuracy. However, upon basic online search, I found that we'll need a frame synchronized rig for such purpose. I also found that usually LiDAR devices have a sync port for such purpose, but I don't know how I can connect it with a camera so that both devices generate frames capturing the exact same instant in time? Do usual cameras have such a port or do I need to buy a special camera?

Thanks.

$\endgroup$

2 Answers 2

1
$\begingroup$

However, upon basic online search, I found that we'll need a frame synchronized rig for such purpose

This is not a simple problem and there exist several solutions, based on project needs, to achieve what you are trying to do. A basic search isn't enough. This forum is for specific questions / errors you want resolved.

As a starting point, you can take a look at Autoware's object detection. They have a pre-trained network you can use as a starting point. If you do not use a triggered camera, you may co-calibrate your lidar and camera (again autoware has one) to get transforms between them and use that as an input to your network.

You could also take a look at lidar-bonnetal who use range images projected from a lidar. Maybe you can incorporate color information in them?

You will also find several newer works on ArXiV like RoIFusion, end-to-end approach, etc.

$\endgroup$
1
$\begingroup$

As your application does not need on-the-fly sync between the devices, running an SW-based time lag estimation is also one way to do it.

The following materials might help to start with.

https://github.com/ethz-asl/kalibr

Spatiotemporal camera-LiDAR calibration: A targetless and structureless approach, C Park, P Moghadam, S Kim, S Sridharan, C Fookes

But please be aware that this is one of the most difficult problems when you use a multi-modal sensor.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.