0
$\begingroup$

Rosanswers logo

Hello everybody,

I am a relatively new ROS user (but I've spent years working with other middlewares), and up to now I am generally happy about it. I have a doubt related to my software framework I would like to clarify. To this end, I will describe the framework I have, so that some more experienced ROS user can tell me where I am doing wrong:

  • I have a node that is composed of two controllers (one for each of the Baxter's arms). Each controller subscribes to a variety of topics and publishes to a variety of topics. They are implemented as classes, and they both use an AsyncSpinner.

  • This is a pseudo-code of my framework. In the main:

     int main()
     {
         Controller c1("left");
         Controller c2("right");
    
         ros::spin();
     }
    

    in the constructors:

     Controller::Controller(string limb): _limb(limb), spinner(4)
     {
         ...
    
         spinner.start();
     }
    

Question 1: is this a correct initialization of an AsyncSpinner? Right now I have two AsyncSpinners that are initialized in the same process/node. Question 2: Do they share the same 4 threads? I am not using custom callback queues.

  • A part from subscribing to topics, both controllers have a main control loop that I implemented as a pthread, which runs in parallel with everything else. It is crucial for my pthreads to have a constant control rate of at least 100Hz, but at the same time I would like to update the information coming from the subscribers while the thread is doing its job. What I did to this end was the following pseudo code related to the run() of the thread:

     pthread_run()
     {
         while(ros::ok())
         {
             ...
    
             ros::spinOnce();
             ros::Rate(100).sleep();
         }
     }
    

Question 3: with the presence of an AsyncSpinner, is the ros::spinOnce() necessary, or can I remove it? As far as I understood I should not need a ros::spinOnce(), because my callback queue is guaranteed to be emptied in the other three threads I fed the spinner with. Is it correct?

  • No matter what I try, some times I experience dropped frames in my pthreads, which negatively affect my control loop. The reasons for this might be multiple (delays in the network, etc), but here is my Question 4: is this issue related to how I implemented my code? As a matter of fact, this is the main reason for which created this question. I am sure it is not a CPU load problem.
  • I was thinking of dropping altogether the pthreads and implementing the control loops as callbacks that I would then add to the ros::getGlobalCallbackQueue() so that I do not have to take care of the threading at all. Question 5: does this make any sense, or is it going to solve my problem?

Thank you for your help. I know that some of this questions might seems stupid, but I didn't find anything useful online, and I would like to use ROS to the best of its capabilities.

Answer to kramer: thank you for your detailed explaination. I ran a bit of testing in the past few days, and here is my report (plus some answers to your questions).

  1. With dropped frames I mean exactly as you said, i.e. that the loop runs slower than 100hz. The problem is not necessarily related to the fact that sometimes I have 20ms delays, but it is mainly due to the fact that sometimes I have delays as big as 300ms or even 1 second, which I cannot afford in my control loop.
  2. I'm measuring delays only in my pthread control loop (the only part I care about this). Plus, my callbacks theoretically should be always faster than 10ms since I only assign some internal variables (doubles, etc). Also, the ros::Rate is only in my pthread_run() function, and not in the callbacks.
  3. I never wait for messages to come, as I am always saving the last one that I received from the topics I subscribed to.
  4. I removed the spinOnce(), and it did not affect the code. Thanks for clarifying that to me!
  5. I changed my code in the pthread_run() as you explained. Also this one did not affect my controllers. I honestly don't understand why the creation of the object ros::Rate r(100) inside the loop should slow down things: is there something else I am missing?
  6. I tried to instantiate a MultiThreadedSpinner into the main() function and get rid of the AsyncSpinners as you suggested, but that solution didn't work. I think it does not apply to my case because my pthread_run() are triggered from a callback the controllers subscribe to (basically, other nodes can request actions to the controllers, which then trigger the pthread if needed).
  7. I will take a look at the Timers. Thank you for that, I was not aware of them. But as you said, I don't think that my problem would be solved by using them. FYI the most complex thing my controllers do is publishing joint configurations on a topic.

Originally posted by alecive on ROS Answers with karma: 3 on 2016-08-11

Post score: 0

$\endgroup$

1 Answer 1

0
$\begingroup$

Rosanswers logo

I think this is much more complicated than it needs to be, particularly since you're not using custom callback queue(s). As I see it, the key concept to latch onto here is that all received messages go through a callback queue, each processed when a spin occurs. The primary simplifying factor here is that you only have a single callback queue (the global callback queue), with the result that a single ros::spin(), MultiThreadedSpinner::spin(), or AsyncSpinner.start() will process all queued messages as quickly as possible (that is, when a thread is available; not spinOnce).

One minor digression: it's not clear to me what dropped frames means here (in the pthreads loop). I'm guessing it means the loop sometimes runs slower than 100Hz, but it could mean you're occasionally dropping or missing message data. I think either might be the case:

  1. The former makes some sense to me, as a spinOnce will block while a callback is performed; if the callback takes longer than 10ms, it'll throw off your timing. Actually, as your code is written, I'm pretty sure it will throw off you timing, since you instantiate a new ros::Rate inside the loop. If I have this right, the time for a given loop iteration will be the sum of the time to process a callback, assuming there is one, plus a 10ms sleep.
  2. The latter also seems possible to me, as a callback handled by one of the AsyncSpinner threads in its entirety while blocked on the spinOnce might appear to be lost (depending on your data handling).

Pulling back out of the digression, it seems to me you might be better served by using a MultiThreadedSpinner, removing the AsyncSpinner from the Controller class, like so:

int main() {
    Controller c1("left");
    Controller c2("right");
    // start controller threads
    ros::MultiThreadedSpinner spinner(8); // 8 = 4 per controller
    spinner.spin();
}

The Controller class can retain its thread (which would need to be started prior to calling spinner.spin()), but modified like so:

pthread_run() {
    ros::Rate r(100);
    while(ros::ok()) {
        ...
        r.sleep();
    }
}

You might also want to look at (recommended) using Timers instead of ros::Rate (which utilize the callback queue -- see (5) below). Now, to your questions:

  1. Yes, correct initialization (as described, maybe not as desired)
  2. No, they use different threads
  3. The spinOnce can be removed
  4. I think item (1) in my discussion of dropped frames is the issue; if so, then yes, it's your implementation
  5. Since I don't know what is in your control loops, I don't know if it makes sense to put that functionality in callbacks. But it's plausible -- for instance, it could be done with a Timer's callback method.

Honestly, though, I think just moving the ros::Rate instantiation outside the loop will solve your most important issue.


Originally posted by kramer with karma: 1470 on 2016-08-11

This answer was ACCEPTED on the original site

Post score: 2


Original comments

Comment by alecive on 2016-08-17:
Hello kramer, thank you for the reply! I answered to you by editing my first post.

Comment by kramer on 2016-08-17:
I'll try to get back to this later, but I wanted to explain: having the ros::Rate instantiation inside the loop will always sleep ~10ms; having it outside the loop will cause the sleep to be ~10ms minus the prior time in the loop. Inside: 10ms+callback (callback may be ~0); outside: ~10ms.

Comment by kramer on 2016-08-17:
I should say that I expected there to be maybe an extra 10ms in your loop (dependent, of course, on what the ... represents). I didn't expect up to 1s delays, which seems...odd, to say the least.

Comment by kramer on 2016-08-19:
Without seeing actual code, I'm not able to add much. I now think your pthread execution operates differently than I expected, so can't help more.

Comment by alecive on 2016-08-19:
kramer, sorry for the delay: I wanted to perform some more testing before coming back to you. I'm starting to think that the 1s delays are caused by spikes in the network usage that top up the bandwidth. I am changing my code in order to address this. I'll answer you when I will have more info.

Comment by alecive on 2016-12-05:
@kramer , I dug deeper into the issue. Apparently, it was not an issue in my code, nor the network: rather, the service request I was asking (that is, an IK call for the Baxter robot), some times got stuck into its computation. Changing the IK solver solved the issue altogether. Thank you!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.