0
$\begingroup$

Rosanswers logo

I am running a multi robot setup in simulation. Most of the times gazebo runs fine, but occasionally it will crash out (about 1 in 3 times). The relevant output of the launch script looks like this:

Gazebo multi-robot simulator, version 1.0.2
Copyright (C) 2011 Nate Koenig, John Hsu, Andrew Howard, and contributors.
Released under the Apache 2 License.
http://gazebosim.org

Msg Waiting for master
Msg Connected to gazebo master @ http://localhost:11345
Msg Waiting for master[ INFO] [1346872005.741139817]: waitForService: Service [/gazebo/set_physics_properties] has not been advertised, waiting...

Msg Connected to gazebo master @ http://localhost:11345
gzserver: /usr/include/boost/smart_ptr/shared_ptr.hpp:418: T* boost::shared_ptr<T>::operator->() const [with T = gazebo::transport::Publisher]: Assertion `px != 0' failed.
Service call failed: transport error completing service call: unable to receive data from sender, check sender's logs for details
Service call failed: transport error completing service call: unable to receive data from sender, check sender's logs for details
Aborted (core dumped)
[ens32_world-2] process has died [pid 6080, exit code 134, cmd optirun /opt/ros/fuerte/stacks/simulator_gazebo/gazebo/scripts/gazebo /home/piyushk/ros/experimental/stacks/bwi/bwi_gazebo/worlds/ens32.world __name:=ens32_world __log:=/home/piyushk/.ros/log/ce555632-f78c-11e1-888f-848f69a9524e/ens32_world-2.log].
log file: /home/piyushk/.ros/log/ce555632-f78c-11e1-888f-848f69a9524e/ens32_world-2*.log
[person1/spawn_robot-23] process has finished cleanly
log file: /home/piyushk/.ros/log/ce555632-f78c-11e1-888f-848f69a9524e/person1-spawn_robot-23*.log
[segbot1/spawn_robot-6] process has finished cleanly
log file: /home/piyushk/.ros/log/ce555632-f78c-11e1-888f-848f69a9524e/segbot1-spawn_robot-6*.log
Error [Transport.cc:102] Unable to get topic namespaces in [50] tries.
Error [Node.cc:68] No namespace found
terminate called after throwing an instance of 'std::length_error'
  what():  basic_string::_S_create
/opt/ros/fuerte/stacks/simulator_gazebo/gazebo/scripts/gui: line 2:  6892 Aborted                 (core dumped) `rospack find gazebo`/gazebo/bin/gzclient -g `rospack find gazebo`/lib/libgazebo_ros_paths_plugin.so
[gazebo_gui-3] process has died [pid 6082, exit code 134, cmd optirun /opt/ros/fuerte/stacks/simulator_gazebo/gazebo/scripts/gui __name:=gazebo_gui __log:=/home/piyushk/.ros/log/ce555632-f78c-11e1-888f-848f69a9524e/gazebo_gui-3.log].
log file: /home/piyushk/.ros/log/ce555632-f78c-11e1-888f-848f69a9524e/gazebo_gui-3*.log
^C[segbot2/move_base-22] killing on exit
[segbot2/fake_localization-21] killing on exit
[segbot2/map_tf_provider-20] killing on exit
[segbot2/kinect_laser_narrow-19] killing on exit
[segbot2/kinect_laser-18] killing on exit
 [segbot2/pointcloud_throttle-17] killing on exit
[segbot2/openni_manager-16] killing on exit
[segbot2/spawn_robot-15] killing on exit
[segbot2/robot_state_publisher-14] killing on exit
[segbot1/move_base-13] killing on exit
[segbot1/fake_localization-12] killing on exit
[segbot1/map_tf_provider-11] killing on exit
[segbot1/kinect_laser_narrow-10] killing on exit
[segbot1/kinect_laser-9] killing on exit
[segbot1/pointcloud_throttle-8] killing on exit
[segbot1/openni_manager-7] killing on exit
[segbot1/robot_state_publisher-5] killing on exit
[bwi_map_server-4] killing on exit
Traceback (most recent call last):
  File "/home/piyushk/ros/experimental/stacks/bwi/bwi_apps/scripts/bwi_map_server", line 216, in <module>
    MapServer()
  File "/home/piyushk/ros/experimental/stacks/bwi/bwi_apps/scripts/bwi_map_server", line 151, in __init__
    rate.sleep();
  File "/opt/ros/fuerte/lib/python2.7/dist-packages/rospy/timer.py", line 78, in sleep
    sleep(self.sleep_dur - elapsed)
  File "/opt/ros/fuerte/lib/python2.7/dist-packages/rospy/timer.py", line 133, in sleep
    raise rospy.exceptions.ROSInterruptException("ROS shutdown request")
rospy.exceptions.ROSInterruptException: ROS shutdown request
Traceback (most recent call last):
  File "/opt/ros/fuerte/stacks/simulator_gazebo/gazebo/scripts/spawn_model", line 277, in <module>
    sm.callSpawnService()
  File "/opt/ros/fuerte/stacks/simulator_gazebo/gazebo/scripts/spawn_model", line 246, in callSpawnService
    success = gazebo_interface.spawn_urdf_model_client(self.model_name, model_xml, self.robot_namespace, initial_pose, self.reference_frame, self.gazebo_namespace)
  File "/opt/ros/fuerte/stacks/simulator_gazebo/gazebo/src/gazebo/gazebo_interface.py", line 28, in spawn_urdf_model_client
    rospy.wait_for_service(gazebo_namespace+'/spawn_urdf_model')
  File "/opt/ros/fuerte/lib/python2.7/dist-packages/rospy/impl/tcpros_service.py", line 158, in wait_for_service
    raise ROSInterruptException("rospy shutdown")
rospy.exceptions.ROSInterruptException: rospy shutdown
[rosout-1] killing on exit
[master] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done

The gazebo log file does not exist. Additionally the log files of the three spawn scripts are below:

person1-spawn_robot-23.log

[rospy.client][INFO] 2012-09-05 14:06:36,932: init_node, name[/person1/spawn_robot], pid[6696]
[xmlrpc][INFO] 2012-09-05 14:06:36,932: XML-RPC server binding to 0.0.0.0
[xmlrpc][INFO] 2012-09-05 14:06:36,932: Started XML-RPC server [http://piyushk-laptop:42536/]
[rospy.impl.masterslave][INFO] 2012-09-05 14:06:36,932: _ready: http://piyushk-laptop:42536/
[rospy.init][INFO] 2012-09-05 14:06:36,933: ROS Slave URI: [http://piyushk-laptop:42536/]
[xmlrpc][INFO] 2012-09-05 14:06:36,933: xml rpc node: starting XML-RPC server
[rospy.registration][INFO] 2012-09-05 14:06:36,933: Registering with master node http://localhost:11311
[rospy.init][INFO] 2012-09-05 14:06:37,033: registered with master
[rospy.rosout][INFO] 2012-09-05 14:06:37,034: initializing /rosout core topic
[rospy.rosout][INFO] 2012-09-05 14:06:37,036: connected to core topic /rosout
[rospy.simtime][INFO] 2012-09-05 14:06:37,037: initializing /clock core topic
[rospy.simtime][INFO] 2012-09-05 14:06:37,038: connected to core topic /clock
[rosout][INFO] 2012-09-05 14:06:37,039: waiting for service /ens32_world/spawn_urdf_model
[rospy.internal][INFO] 2012-09-05 14:06:37,332: topic[/rosout] adding connection to [/rosout], count 0
[rospy.core][INFO] 2012-09-05 14:06:46,559: signal_shutdown [atexit]
[rospy.internal][INFO] 2012-09-05 14:06:46,566: topic[/rosout] removing connection to /rosout
[rospy.impl.masterslave][INFO] 2012-09-05 14:06:46,566: atexit

segbot1-spawn_robot-6.log:

[rospy.client][INFO] 2012-09-05 14:06:36,019: init_node, name[/segbot1/spawn_robot], pid[6106]
[xmlrpc][INFO] 2012-09-05 14:06:36,019: XML-RPC server binding to 0.0.0.0
[xmlrpc][INFO] 2012-09-05 14:06:36,019: Started XML-RPC server [http://piyushk-laptop:42289/]
[rospy.init][INFO] 2012-09-05 14:06:36,019: ROS Slave URI: [http://piyushk-laptop:42289/]
[rospy.impl.masterslave][INFO] 2012-09-05 14:06:36,019: _ready: http://piyushk-laptop:42289/
[rospy.registration][INFO] 2012-09-05 14:06:36,020: Registering with master node http://localhost:11311
[xmlrpc][INFO] 2012-09-05 14:06:36,020: xml rpc node: starting XML-RPC server
[rospy.init][INFO] 2012-09-05 14:06:36,120: registered with master
[rospy.rosout][INFO] 2012-09-05 14:06:36,120: initializing /rosout core topic
[rospy.rosout][INFO] 2012-09-05 14:06:36,123: connected to core topic /rosout
[rospy.simtime][INFO] 2012-09-05 14:06:36,124: initializing /clock core topic
[rospy.simtime][INFO] 2012-09-05 14:06:36,126: connected to core topic /clock
[rosout][INFO] 2012-09-05 14:06:36,130: waiting for service /ens32_world/spawn_urdf_model
[rospy.internal][INFO] 2012-09-05 14:06:36,317: topic[/rosout] adding connection to [/rosout], count 0
[rospy.internal][INFO] 2012-09-05 14:06:46,406: topic[/clock] adding connection to [http://piyushk-laptop:33181/], count 0
[rospy.internal][WARNING] 2012-09-05 14:06:46,560: Unknown error initiating TCP/IP socket to piyushk-laptop:58661 (http://piyushk-laptop:33181/): Traceback (most recent call last):
  File "/opt/ros/fuerte/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py", line 511, in connect
    self.socket.connect((dest_addr, dest_port))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

[rospy.core][INFO] 2012-09-05 14:06:46,560: signal_shutdown [atexit]
[rospy.internal][INFO] 2012-09-05 14:06:46,569: topic[/rosout] removing connection to /rosout
[rospy.impl.masterslave][INFO] 2012-09-05 14:06:46,569: atexit

segbot2-spawn_robot-15.log:

[rospy.client][INFO] 2012-09-05 14:06:36,570: init_node, name[/segbot2/spawn_robot], pid[6382]
[xmlrpc][INFO] 2012-09-05 14:06:36,570: XML-RPC server binding to 0.0.0.0
[xmlrpc][INFO] 2012-09-05 14:06:36,571: Started XML-RPC server [http://piyushk-laptop:39168/]
[rospy.init][INFO] 2012-09-05 14:06:36,571: ROS Slave URI: [http://piyushk-laptop:39168/]
[rospy.impl.masterslave][INFO] 2012-09-05 14:06:36,571: _ready: http://piyushk-laptop:39168/
[xmlrpc][INFO] 2012-09-05 14:06:36,571: xml rpc node: starting XML-RPC server
[rospy.registration][INFO] 2012-09-05 14:06:36,572: Registering with master node http://localhost:11311
[rospy.init][INFO] 2012-09-05 14:06:36,671: registered with master
[rospy.rosout][INFO] 2012-09-05 14:06:36,672: initializing /rosout core topic
[rospy.rosout][INFO] 2012-09-05 14:06:36,675: connected to core topic /rosout
[rospy.simtime][INFO] 2012-09-05 14:06:36,676: initializing /clock core topic
[rospy.simtime][INFO] 2012-09-05 14:06:36,677: connected to core topic /clock
[rosout][INFO] 2012-09-05 14:06:36,681: waiting for service /ens32_world/spawn_urdf_model
[rospy.internal][INFO] 2012-09-05 14:06:36,931: topic[/rosout] adding connection to [/rosout], count 0
[rospy.internal][INFO] 2012-09-05 14:06:46,404: topic[/clock] adding connection to [unknown], count 0
[rospy.internal][WARNING] 2012-09-05 14:06:46,561: Unknown error initiating TCP/IP socket to piyushk-laptop:58661 (http://piyushk-laptop:33181/): Traceback (most recent call last):
  File "/opt/ros/fuerte/lib/python2.7/dist-packages/rospy/impl/tcpros_base.py", line 511, in connect
    self.socket.connect((dest_addr, dest_port))
  File "/usr/lib/python2.7/socket.py", line 224, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 111] Connection refused

[rospy.core][INFO] 2012-09-05 14:07:23,378: signal_shutdown [signal-2]
[rospy.internal][INFO] 2012-09-05 14:07:23,389: topic[/rosout] removing connection to /rosout
[rospy.impl.masterslave][INFO] 2012-09-05 14:07:23,390: signal-2
[rospy.core][INFO] 2012-09-05 14:07:24,039: signal_shutdown [atexit]

Update #1

Here is the output with gdb for this error.

Here is another error that was produced while running gdb, though I don't remember seeing it before. I was unable to reproduce it after this run.

Update #2

Output after changing GZ_REGISTER_SYSTEM_PLUGIN. The gazebo trunk was locally downloaded for this run (r40064)

Update #3

Output Debug set for ROS_BUILD_TYPE.


Originally posted by piyushk on ROS Answers with karma: 2871 on 2012-09-05

Post score: 0


Original comments

Comment by hsu on 2012-09-06:
one way to begin the debugging process is to start gazebo with gdb attached (i.e. roslaunch gazebo_worlds debug.launch) and look at the backtrace of gzserver when it segfaults.

Comment by piyushk on 2012-09-06:
@hsu: see end of updated question.

$\endgroup$

1 Answer 1

0
$\begingroup$

Rosanswers logo

Hi Piyushk, thanks for the backtraces, can you put the GZ_REGISTER_SYSTEM_PLUGIN macro directly into gazebo_ros_api_plugin.cpp? i.e. at the end of the file, replace:

// Register this plugin with the simulator
GZ_REGISTER_SYSTEM_PLUGIN(GazeboRosApiPlugin)
}

with

extern "C" gazebo::SystemPlugin *RegisterPlugin();
gazebo::SystemPlugin *RegisterPlugin()
{
  return new GazeboRosApiPlugin();
}

and see if the bt gives us some more detail?

thanks, John


Originally posted by hsu with karma: 5780 on 2012-09-06

This answer was ACCEPTED on the original site

Post score: 0


Original comments

Comment by piyushk on 2012-09-06:
John, I have updated the question. I cannot spot a difference. the simulator_gazebo trunk was used instead of simulator_gazebo-1.6.14 deb

Comment by hsu on 2012-09-06:
thanks. what on line /home/piyushk/ros/simulator_gazebo/gazebo/src/gazebo_ros_api_plugin.cpp:1948?

Comment by piyushk on 2012-09-06:
I now see what you are driving at. Unfortunately, it was the same line as before (i.e. namespace closing brace). I recompiled gazebo with ROS_BUILD_TYPE set as Debug, and I believe more info is now available. I have updated the question with the gdb output.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.