Recording Large Rosbags of Lidar & Camera Data

Question

I have been trying to find some documentation on this but was unable to really see anyhting helpful.

We have 5 cameras and 3 Lidars and need to record data from all sensors. For now we have been recording only two cameras (Allied Vision G-319C) and one Lidar (Velodyne VLP-32C) and are already running into problems.

The data is published at 20 Hz (lidar) and 30 Hz (cameras). Recording only one image for each camera in compressed format and the /velodyne_points results in the buffer being overflown and frames/points being dropped. The data is recorded to an internal SSD but I also tried an external SSD which works slightly worse.

I am wondering what the best practice would be in this case such as recording multiple rosbags at the same time, for example record one rosbag for each camera or something like that to distribute the load - or not even using rosbags but some other tool that i am unaware of.

(I have played around with different buffer sizes (setting it to 0 results in the rosbag automatically shutting down quickly) and splitting the rosbags at different sizes (500 MB, 5000 MB) without much improvement.)

Hardware: Astuff Spectra/Neuosys 6109GC with 32 GB of ram, two internal 265 GB and 1 TB SSDs, external 2 TB SSD, XEON E2278G 8th Gen

Thanks

---

Originally posted by JoloGermanAg on ROS Answers with karma: 123 on 2022-06-09

Post score: 0

Answer 1

There are a couple tricks out there that can be used:

• Do you really need full frame rate for the sensors? If not, you can insert something like the throttle node from topic_tools in between your actual sensors and the rosbag recorder to downsample the frame rate to a more manageable amount of data.
• I've personally always had better luck compressing the bagfiles as a separate step AFTER recording (so, record uncompressed, then when the bagfile finishes or splits you compress it).

---

Originally posted by fergs with karma: 13902 on 2022-06-10

This answer was NOT ACCEPTED on the original site

Post score: 2

---

Original comments

Comment by JoloGermanAg on 2022-06-13:
I suppose you are right in that it's not necessary to record at the full frequency - i have tested this and it does work at around 10 Hz for everything. I also realized that our images are rather large so resizing them may help too. Thank you for your answer!

Answer 2

We are recording our sensor data (3 x 3D LiDAR, 3 x 2k rgb camera, IMU, GPS) in the following way and not running into any issues:

rosbag record -O "$NAME"_"$cnt" --split --size=4096 /topics

Although we had some problems (frames dropping) with our network card of the computer since our cameras and lidars are ethernet based. We bought a 10Gbe to thunderbolt 4 adapter and all is wokring fine that way

---

Originally posted by GeorgNo with karma: 183 on 2022-06-13

This answer was NOT ACCEPTED on the original site

Post score: 2

---

Original comments

Comment by JoloGermanAg on 2022-06-13:
Thank you, I will try this as well!

Answer 3

The solution for our problem was to reduce the image sizes using image_proc/resize. We are now able to record at full frequency. Thanks for all your answers!

---

Originally posted by JoloGermanAg with karma: 123 on 2022-06-20

This answer was NOT ACCEPTED on the original site

Post score: 2