0
$\begingroup$

Rosanswers logo

I am trying to increase the responsiveness of a control loop. I am controlling the pan/tilt servos of a camera (attached to a Raspberry Pi 3) based on the center of a bounding box found by YOLO vision system (running on a desktop pc with GTX 1080 ti GPU). I am observing very jumpy servo movements.

So:

 Yolo(on pc)   <----wifi------ PiCam(Pi3)

 PID(on pc)    -----wifi-----> ServoController(Pi3)

The desired servo positions are sent as messages by the PID controller on the pc at 10 Hz (this is confirmed by the messages seen on rqt_console and rostopic Hz). The message queue size of the publisher is 1.

On the Pi3 I have set the loop rate of the servo controller to values between 10 and 50 Hz. I have set the subscriber queue size to 1 and 10.

  • With a ServoController queue size of 1, rqt_console shows only about one in 10 of the control messages are processed. A control signal sent to the servos at 1 or 2 Hz resulting in very jumpy movements.
  • With a ServoController queue size of 10, rqt_console shows that all the control messages are processed, but they are processed very quickly in blocks of about 10, then nothing happens for almost a second, then another block of 10 are processed. The servos remain very jumpy because all the small incremental movements are processed at the same time and the largest step dominates.

My conclusion is that the controller on the Pi3 is processing in bursts, but why ? Changing the loop rate from 10 to 50 did not improve matters. I checked the cpu utilisation on the Pi3 but it is never more than 7%.

Here is output from rqt_console which shows the bursts. 'I heard' denotes a message processed by the servo controller. PID denotes a message sent from the PID loop.


    "I heard: [14, 99]",/PiServoNode,1559747410.29711
    "I heard: [14, 101]",/PiServoNode,1559747410.29413
    "I heard: [14, 124]",/PiServoNode,1559747410.29118
    "I heard: [14, 113]",/PiServoNode,1559747410.2879
    "PID X: pan=99, err=84, cumErr=204, P=92.400000 I=12.240000, clamp=10000
    ",/eye_control_node,1559747409.25256
    "PID X: pan=101, err=91, cumErr=120, P=100.100000 I=7.200000, clamp=10000
    ",/eye_control_node,1559747409.15253
    "PID X: pan=124, err=118, cumErr=29, P=129.800000 I=1.740000, clamp=10000
    ",/eye_control_node,1559747409.05254
    "PID X: pan=113, err=114, cumErr=-89, P=125.400000 I=-5.340000, clamp=10000
    ",/eye_control_node,1559747408.95256
    "I heard: [14, 101]",/PiServoNode,1559747409.1169
    "I heard: [14, 91]",/PiServoNode,1559747409.11382
    "I heard: [14, 78]",/PiServoNode,1559747409.11077
    "I heard: [14, 65]",/PiServoNode,1559747409.10774
    "I heard: [14, 54]",/PiServoNode,1559747409.10459
    "I heard: [14, 43]",/PiServoNode,1559747409.1008
    "I heard: [14, 31]",/PiServoNode,1559747409.09732
    "I heard: [14, 20]",/PiServoNode,1559747409.09406
    "I heard: [14, 13]",/PiServoNode,1559747409.09102
    "I heard: [14, 18]",/PiServoNode,1559747409.0879
    "PID X: pan=101, err=109, cumErr=-203, P=119.900000 I=-12.180000, clamp=10000
    ",/eye_control_node,1559747408.85256
    "PID X: pan=91, err=106, cumErr=-312, P=116.600000 I=-18.720000, clamp=10000
    ",/eye_control_node,1559747408.75254
    "PID X: pan=78, err=100, cumErr=-418, P=110.000000 I=-25.080000, clamp=10000
    ",/eye_control_node,1559747408.65253
    "PID X: pan=65, err=93, cumErr=-518, P=102.300000 I=-31.080000, clamp=10000
    ",/eye_control_node,1559747408.55253
    "PID X: pan=54, err=88, cumErr=-611, P=96.800000 I=-36.660000, clamp=10000
    ",/eye_control_node,1559747408.45256
    "PID X: pan=43, err=82, cumErr=-699, P=90.200000 I=-41.940000, clamp=10000
    ",/eye_control_node,1559747408.35254
    "PID X: pan=31, err=75, cumErr=-781, P=82.500000 I=-46.860000, clamp=10000
    ",/eye_control_node,1559747408.25256
    "PID X: pan=20, err=69, cumErr=-856, P=75.900000 I=-51.360000, clamp=10000
    ",/eye_control_node,1559747408.15256
    "PID X: pan=13, err=66, cumErr=-925, P=72.600000 I=-55.500000, clamp=10000
    ",/eye_control_node,1559747408.05259
    "PID X: pan=18, err=75, cumErr=-991, P=82.500000 I=-59.460000, clamp=10000
    ",/eye_control_node,1559747407.95258
    "PID X: pan=59, err=119, cumErr=-1066, P=130.900000 I=-63.960000, clamp=10000
    ",/eye_control_node,1559747407.85262
    "PID X: pan=46, err=113, cumErr=-1185, P=124.300000 I=-71.100000, clamp=10000
    ",/eye_control_node,1559747407.75258
    "I heard: [14, 33]",/PiServoNode,1559747407.91607
    "I heard: [14, 19]",/PiServoNode,1559747407.91308
    "I heard: [14, 5]",/PiServoNode,1559747407.91008
    "I heard: [14, -4]",/PiServoNode,1559747407.90697
    "I heard: [14, -21]",/PiServoNode,1559747407.90379
    "I heard: [14, -32]",/PiServoNode,1559747407.90056
    "I heard: [14, -47]",/PiServoNode,1559747407.89724
    "I heard: [14, -60]",/PiServoNode,1559747407.89402
    "I heard: [14, -68]",/PiServoNode,1559747407.89103
    "I heard: [14, -74]",/PiServoNode,1559747407.88791
    "PID X: pan=33, err=107, cumErr=-1298, P=117.700000 I=-77.880000, clamp=10000
    ",/eye_control_node,1559747407.65254
    "PID X: pan=19, err=100, cumErr=-1405, P=110.000000 I=-84.300000, clamp=10000
    ",/eye_control_node,1559747407.55253
    "PID X: pan=5, err=92, cumErr=-1505, P=101.200000 I=-90.300000, clamp=10000
    ",/eye_control_node,1559747407.45258
    "PID X: pan=-4, err=88, cumErr=-1597, P=96.800000 I=-95.820000, clamp=10000
    ",/eye_control_node,1559747407.35254
    "PID X: pan=-21, err=77, cumErr=-1685, P=84.700000 I=-101.100000, clamp=10000
    ",/eye_control_node,1559747407.25254
    "PID X: pan=-32, err=70, cumErr=-1762, P=77.000000 I=-105.720000, clamp=10000
    ",/eye_control_node,1559747407.15258
    "PID X: pan=-47, err=60, cumErr=-1832, P=66.000000 I=-109.920000, clamp=10000
    ",/eye_control_node,1559747407.05256
    "PID X: pan=-60, err=51, cumErr=-1892, P=56.100000 I=-113.520000, clamp=10000
    ",/eye_control_node,1559747406.95257
    "PID X: pan=-68, err=46, cumErr=-1943, P=50.600000 I=-116.580000, clamp=10000
    ",/eye_control_node,1559747406.85253
    "PID X: pan=-74, err=43, cumErr=-1989, P=47.300000 I=-119.340000, clamp=10000
    ",/eye_control_node,1559747406.75252
    "PID X: pan=-75, err=45, cumErr=-2032, P=49.500000 I=-121.920000, clamp=10000
    ",/eye_control_node,1559747406.65252
    "PID X: pan=-81, err=41, cumErr=-2077, P=45.100000 I=-124.620000, clamp=10000
    ",/eye_control_node,1559747406.5525


Originally posted by elpidiovaldez on ROS Answers with karma: 142 on 2019-06-05

Post score: 0


Original comments

Comment by gvdhoorn on 2019-06-05:\

My conclusion is that the controller on the Pi3 is processing in bursts, but why ?

I believe there are many older Q&As already discussing this. See #q303464, #q237592, #q272375 and #q290122. Did you not find those when searching for "messages bursts"?

The first thing to check is whether this is due to Nagle.

Comment by elpidiovaldez on 2019-06-05:
No I did not find those, but my original searches (and the original title of my post) used the word 'chunks' instead of 'bursts'. I edited it later when I thought of a better word ! Thanks for a great pointer, I had never come across any of that material in the ROS tutorials that I have read.

$\endgroup$

1 Answer 1

0
$\begingroup$

Rosanswers logo

I am closing the post because I have now found a bug in the ServoController code on the Pi3 that was reducing its rate to 2 Hz, so the problem was nothing to do with ROS. I am also now using the 'tcpNoDelay' and 'unreliable' transport options which are described in the documents linked by gvdhoorn. These are just what I need, so some good came out of this.


Originally posted by elpidiovaldez with karma: 142 on 2019-06-05

This answer was ACCEPTED on the original site

Post score: 0

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.