Are Rospy subscriber callbacks executed sequentially for a single topic?

Question

Suppose a ROS node receives new batch of data on a topic while the previous batch is still being processed by callback. Will ROS wait for its completion or will it spin a new thread and run both at the same time?

I'm asking because if they are sequential there is no need for a lock, but if they're not I'd need to use locking inside the callbacks.

I looked up the documentation for the rospy.Subcriber class, but it has no mention of this.

Answer 1

I prototyped a bit to investigate this behaviour. Ran the following node:

#!/usr/bin/env python                                                                                                                                                                                       
import rospy                                                                                                                                                                                                
from threading import Lock                                                                                                                                                                                  
from std_msgs.msg import String                                                                                                                                                                             
mutex = Lock()                                                                                                                                                                                              
def handler(data):                                                                                                                                                                                          
    print "Handle called"                                                                                                                                                                                   
    mutex.acquire()                                                                                                                                                                                         
    print "Handle executed"                                                                                                                                                                                 
    import time; time.sleep(10)                                                                                                                                                                              
    mutex.release()                                                                                                                                                                                         
rospy.init_node('test')                                                                                                                                                                                     
rospy.Subscriber('/foo', String, handler)                                                                                                                                                                   
rospy.spin()

When I publish several individual messages I see that indeed the handler is getting called on another thread while the current handler is still in execution.

rospy pub /foo std_msgs/String "data: 'hello'"
ctrl-c
rospy pub /foo std_msgs/String "data: 'hello'"
ctrl-c
rospy pub /foo std_msgs/String "data: 'hello'"
ctrl-c

I see the output:

Handle called
Handle executed
Handle called
Handle called
... several seconds later
Handle executed
... several seconds later
Handle executed

However, when I publish a msg periodically from the same publish call:

rospy pub /foo std_msgs/String "data: 'hello'"-r 1 # publish once a second

I see this:

Handle called
Handle executed
... 10 seconds pass
Handle called
Handle executed
... 10 seconds pass
Handle called
Handle executed

I know the message is being published once a second, yet it doesn't spawn a new thread for the callback - so something is throttling the msgs until they have been handled.

So the behavior seems to diff depending on if the msgs are arriving from the same process or multiple processes.

I confirmed that it wasn't just a behaviour of rostopic pub by writing a simple node that publishes a msg every second and it behaves the same.

I haven't looked at the ROS code yet so not sure if its the ROS publisher or subscriber code that is throttling in the same process case.

So if you are publishing a topic from a single source then strictly speaking it seems like you don't need a mutex to protect about re-entry by the same topic handler.

However, in my opinion, considering that you (or a future developer) might call the same work function from a different callback or from a timer or from a main loop, or since you can use the same handler for multiple topics (that might come from different processes) and since adding a mutex is pretty easy and low effort and overhead it seems reasonable.

Answer 2

Apparently the ROS topics have an ambiguous behavior. I have also tested publishing through the rostopic pub and it showed a parallel behavior. But when the publication came from another node it apeears to wait for the execution of the last callback on a sequential approach.

Answer 3

The short answer is yes. In rospy, every subscriber/timer gets its own thread. So for a single topic that means the callback will be executed sequentially. If you are going to be regularly receiving messages before the previous callback has finished, you want to be sure to have reasonable queue size, or you might end up dropping messages!

You're right though, I don't seem to be able to find this documented on the wiki. You can dive into the Topic source code, it looks like the thread on a per-subscriber bases is created here - and in the most basic case it will just start a loop which receives messages and handles callbacks in a receive loop.

Answer 4

It might have something to do with how the ROSMaster helps connect to processes over the network. Two nodes (a publisher and a subscriber) would talk to each other and ROSMaster steps out of the game after they have connected. Even if rosmaster process dies the nodes would continue talking but if one of them dies or has any issue it wouldn't find the other without the master. The master is kind of a lookup table for matching nodes in this regard.

So if you have a "connection" queues might be handled differently. If I subscribe to a publisher with 2 separate processes, It might start a new thread since its a new tcp connection.