0
$\begingroup$

Rosanswers logo

Hi all,

I'm facing a strange problem. While running a custom controller for the whole pr2 joints, the realtime-loop2 die. It happens randomly when a trajectory is sent to the controller. Sometimes at the first trajectory, some other times after the Nth trajectory. After the realtime loop die, the whole robot die. The robot is started with the command roslaunch /etc/ros/robot.launch (and not sudo robot start, i don't know if this makes a difference). He are some logs.

Realtime-loop2:

[roscpp_internal] [2011-04-28 10:17:46,613] [thread 0x7f821803a770]: [DEBUG] TCP socket [58] closed 
[roscpp_internal] [2011-04-28 10:17:49,618] [thread 0x7f820c5d8950]: [DEBUG] Accepted connection on socket [9], new socket [58]
[roscpp_internal] [2011-04-28 10:17:49,618] [thread 0x7f820c5d8950]: [DEBUG] TCPROS received a connection from [10.68.0.1:37309]
[roscpp_internal] [2011-04-28 10:17:49,618] [thread 0x7f820c5d8950]: [DEBUG] Connection: Creating ServiceClientLink for service [/pr2_controller_manager/list_controllers] connected to [callerid=[/arms_controllers_stopper] address=[TCPROS connection to [10.68.0.1:37309 on socket 58]]]
[roscpp_internal] [2011-04-28 10:17:49,618] [thread 0x7f820c5d8950]: [DEBUG] Service    client [/arms_controllers_stopper] wants service [/pr2_controller_manager/list_controllers] with md5sum [39c8d39516aed5c7d76284ac06c220e5]
[roscpp_internal] [2011-04-28 10:17:49,619] [thread 0x7f821803a770]: [DEBUG] TCP socket [58] closed
[roscpp_internal] [2011-04-28 10:17:49,695] [thread 0x7f820c5d8950]: [DEBUG] Accepted connection on socket [9], new socket [58]
[roscpp_internal] [2011-04-28 10:17:49,695] [thread 0x7f820c5d8950]: [DEBUG] TCPROS received a connection from [10.68.0.1:37314]
[roscpp_internal] [2011-04-28 10:17:49,695] [thread 0x7f820c5d8950]: [DEBUG] Connection: Creating ServiceClientLink for service [/l_forearm_cam_trigger/set_waveform] connected to [callerid=[/camera_synchronizer_node] address=[TCPROS connection to [10.68.0.1:37314 on socket 58]]]
[roscpp_internal] [2011-04-28 10:17:49,696] [thread 0x7f820c5d8950]: [DEBUG] Service client [/camera_synchronizer_node] wants service [/l_forearm_cam_trigger/set_waveform] with md5sum [cbb7e900a71a9a437da9999c8d39fff4]
[roscpp_internal] [2011-04-28 10:17:49,696] [thread 0x7f821803a770]: [DEBUG] TCP socket [58] closed
[roscpp_internal] [2011-04-28 10:17:49,697] [thread 0x7f820c5d8950]: [DEBUG] Accepted connection on socket [9], new socket [58]
[roscpp_internal] [2011-04-28 10:17:49,697] [thread 0x7f820c5d8950]: [DEBUG] TCPROS received a connection from [10.68.0.1:37315]
[roscpp_internal] [2011-04-28 10:17:49,697] [thread 0x7f820c5d8950]: [DEBUG] Connection: Creating ServiceClientLink for service [/r_forearm_cam_trigger/set_waveform] connected to [callerid=[/camera_synchronizer_node] address=[TCPROS connection to [10.68.0.1:37315 on socket 58]]]
[roscpp_internal] [2011-04-28 10:17:49,697] [thread 0x7f820c5d8950]: [DEBUG] Service client [/camera_synchronizer_node] wants service [/r_forearm_cam_trigger/set_waveform] with md5sum [cbb7e900a71a9a437da9999c8d39fff4]
[roscpp_internal] [2011-04-28 10:17:49,698] [thread 0x7f821803a770]: [DEBUG] TCP socket [58] closed
[roscpp_internal] [2011-04-28 10:17:49,698] [thread 0x7f820c5d8950]: [DEBUG] Accepted connection on socket [9], new socket [58]
[roscpp_internal] [2011-04-28 10:17:49,698] [thread 0x7f820c5d8950]: [DEBUG] TCPROS received a connection from [10.68.0.1:37316]
[roscpp_internal] [2011-04-28 10:17:49,698] [thread 0x7f820c5d8950]: [DEBUG] Connection: Creating ServiceClientLink for service [/head_camera_trigger/set_waveform] connected to [callerid=[/camera_synchronizer_node] address=[TCPROS connection to [10.68.0.1:37316 on socket 58]]]
[roscpp_internal] [2011-04-28 10:17:49,698] [thread 0x7f820c5d8950]: [DEBUG] Service client [/camera_synchronizer_node] wants service [/head_camera_trigger/set_waveform] with md5sum [cbb7e900a71a9a437da9999c8d39fff4]
[roscpp_internal] [2011-04-28 10:17:49,699] [thread 0x7f820c5d8950]: [DEBUG] Accepted connection on socket [9], new socket [66]
[roscpp_internal] [2011-04-28 10:17:49,699] [thread 0x7f820c5d8950]: [DEBUG] TCPROS received a connection from [10.68.0.1:37317]
[roscpp_internal] [2011-04-28 10:17:49,699] [thread 0x7f820c5d8950]: [DEBUG] Connection: Creating ServiceClientLink for service [/projector_trigger/set_waveform] connected to [callerid=[/camera_synchronizer_node] address=[TCPROS connection to [10.68.0.1:37317 on socket 66]]]
[roscpp_internal] [2011-04-28 10:17:49,699] [thread 0x7f820c5d8950]: [DEBUG] Service client [/camera_synchronizer_node] wants service [/projector_trigger/set_waveform] with md5sum [cbb7e900a71a9a437da9999c8d39fff4]
[roscpp_internal] [2011-04-28 10:17:49,699] [thread 0x7f821803a770]: [DEBUG] TCP socket [58] closed
[roscpp_internal] [2011-04-28 10:17:49,723] [thread 0x7f821803a770]: [DEBUG] TCP socket [66] closed
[ros.pr2_soft_controller] [2011-04-28 10:17:50,513] [thread 0x7f821803a770]: [INFO] New traj received id: 0
[roscpp_internal] [2011-04-28 10:17:52,623] [thread 0x7f820c5d8950]: [DEBUG] Accepted connection on socket [9], new socket [58]
[roscpp_internal] [2011-04-28 10:17:52,624] [thread 0x7f820c5d8950]: [DEBUG] TCPROS received a connection from [10.68.0.1:37334]
[roscpp_internal] [2011-04-28 10:17:52,624] [thread 0x7f820c5d8950]: [DEBUG] Connection: Creating ServiceClientLink for service [/pr2_controller_manager/list_controllers] connected to [callerid=[/arms_controllers_stopper] address=[TCPROS connection to [10.68.0.1:37334 on socket 58]]]
[roscpp_internal] [2011-04-28 10:17:52,624] [thread 0x7f820c5d8950]: [DEBUG] Service client [/arms_controllers_stopper] wants service [/pr2_controller_manager/list_controllers] with md5sum [39c8d39516aed5c7d76284ac06c220e5]
[roscpp_internal] [2011-04-28 10:17:52,624] [thread 0x7f821803a770]: [DEBUG] TCP socket [58] closed
[ros.pr2_soft_controller] [2011-04-28 10:17:55,614] [thread 0x7f821803a770]: [INFO] New traj received id: -1 
(No more logs)

Message from the console:

[realtime_loop-2] process has died [pid 20297, exit code -11].
log files: /u/gmanfred/.ros/log/db5ab696-716f-11e0-a6e6-001517bf0b7b/realtime_loop-2*.log
[ERROR] [1303978676.173134508]: Could not transform imu message from imu_link to base_footprint
[ERROR] [1303978676.655705715]: No data have arrived for more than one second. Assuming that camera is no longer streaming.
[ERROR] [1303978676.676336047]: Could not transform imu message from imu_link to base_footprint
[ERROR] [1303978676.679263734]: No data have arrived for more than one second. Assuming that camera is no longer streaming.
[ERROR] [1303978676.711821408]: No data have arrived for more than one second. Assuming that camera is no longer streaming.
[ERROR] [1303978676.723506736]: No data have arrived for more than one second. Assuming that camera is no longer streaming.
[ERROR] [1303978677.179860108]: Could not transform imu message from imu_link to base_footprint
...

What can make the realtime loop die ? I tryed breaking realtime by overloading it, but it holds on. If a controller crashes, will it cause the realtime loop to die ?

Hope someone can help.

Guido

Originally posted by Guido on ROS Answers with karma: 514 on 2011-04-27

Post score: 1

Improve this question
$\endgroup$

2 Answers 2

Reset to default
0
$\begingroup$

Rosanswers logo

Hi Guido,

If your controller crashes, the entire realtime loop will die. In the PR2 controller manager, all controllers are run in the same process, so if one crashes, the entire process crashes.

Try attaching gdb before loading your controller (you will probably need to run it as root).

Originally posted by sglaser with karma: 649 on 2011-04-28

This answer was NOT ACCEPTED on the original site

Post score: 2

Original comments

Comment by Guido on 2011-04-28:
Thanks for the answer. Regardless of controller, attaching gdb seems to kill the controllers. The realtime loop dies and motors stop. Odom, Imu and narrow stereo complain about missing transforms and services.Gdb: 0x00007fbae6b7956d in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0

Improve this answer
$\endgroup$
0
$\begingroup$

Rosanswers logo

I finally found the problem. As explained in this answer there are various threads to take into account in a controller. In my case, I have two threads to look after: the realtime loop and the one calling callbacks.

As my callback was loading new trajectories in a non concurrent-safe way, it was possible for the realtime loop to read a part of the memory being written by the callback.

I fixed it with a shared pointer, in the same way as the ros controllers do it.

Originally posted by Guido with karma: 514 on 2011-05-01

This answer was ACCEPTED on the original site

Post score: 0

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.