High rates of corrupt bags with rosbag python api?

Question

I have a python action server where the callback creates a rosbag.Bag and then callback subscribing to various messages then write to that bag, and when the action is over the bag is closed and the callbacks then return without attempting to write.

It appears that about 30%-50% of the bags that come out of this say they need to be reindexed, and then after rosbag reindex a huge amount of the messages are lost- what ought to be hundreds of messages are now 10s or even zero- sometimes the reindexed bag is the same size as orig, other times smaller.

---

Originally posted by lucasw on ROS Answers with karma: 8729 on 2018-01-30

Post score: 1

Answer 1

I'm pretty sure that using threading Lock acquire-release around all bag writes will solve this, I'll update this once I have it working (this occurred to me as I was writing the question, but this should be useful for others that make the same mistake).

Note that the rosbag API's are not thread-safe for reading and writing any given bag file. Users of the APIs must ensure no concurrent input/output operations are performed on different threads.

http://wiki.ros.org/rosbag/Code%20API

This appears to be working, haven't seen a corrupt bag yet:

...
from threading import Lock

class Blah:
    def __init__(self):
        self.lock = Lock()
...  
        self.lock.acquire()
        self.bag = rosbag(filename, 'w')
        self.lock.release()
...
        self.lock.acquire()
        self.bag.close()
        self.lock.release()
...
    def foo_callback(self, msg):
        self.lock.acquire()
        self.bag.write("foo", msg)
        self.lock.release()
...

---

Originally posted by lucasw with karma: 8729 on 2018-01-30

This answer was ACCEPTED on the original site

Post score: 4

---

Original comments

Comment by ahendrix on 2018-01-30:
Yeah rqt_bag has similar issues and has a global lock to protect the Bag object.

Comment by codezs09 on 2021-01-07:
Save my day! Thanks for sharing!