ROS2 Windows node always hang

Question

Hi all,

These days I'm having a very strange problem with my application. Since last week it was all good, but this monday I noticed that I can't run more that 2/3 nodes (or ros cli tools) simultaneously. For example: I run 2 nodes, then I do ros2 node list and the cmd doesn't give me any answer and keeps pending. Moreover I can't close it with ctrl+c, but I have to end the task from the task manager. The same goes for ros2 topic list and echo.

Another example is this: I run the first node, then ros2 topic echo .. and I can see the data, then I run the second node and it gets stuck pending.

Just to add some informations, the 2 main nodes work properly togheter, but to develop my application I need to add another node, that with this "state" will never run.

Have you ever faced a similar error? Could it be something about my code?

EDIT: more info

Right now I still have the same problem, but maybe this time I found the cause. I'm writing a node that receive some image from Coppelia, and send it over with a topic. This morning and yesterday I was trying to fix it, because I wasn't able to see the image, but after some "runs" the nodes stopped working. And I still have the same problem as before (third node pending even with the tutorial code).

For the image I set some buffers:

simxUChar *eyeBuffers[2];
dim = resolution[0] * resolution[1] * sizeof(simxUChar);
eyeBuffers[0] = new simxUChar[dim];
eyeBuffers[1] = new simxUChar[dim];

and then set the sensor_msgs.Image msg:

int resolution[2] = { 320, 240 };    
for (int i = 0; i < 2; i++) {
                simxGetVisionSensorImage(clientID, eyes[i], resolution, &eyeBuffers[i], 0, simx_opmode_buffer);
                arr[i] = cv::Mat(resolution[1], resolution[0], CV_8UC3, eyeBuffers[i]);
                flip(arr[i], arr_flip[i], 1);
}
.. and then cv::Bridge to fill the msg. 

I don't have any build errors and I can correctly send and receive the images.

After a few days, I've done some more test and now I can run only 1 node, all the others keep hanging. I thought it was something related to dynamic memory allocation, but I delete[] them.

What I can't understand is why this leads to ROS2 malfunction. I have the impression that something is going to saturate inside ros, because the problem only came out after a few runs (around 20), and before it was all fine and I could run as many nodes as I wanted.

---

Originally posted by alberto on ROS Answers with karma: 100 on 2022-06-07

Post score: 0

---

Original comments

Comment by Joe28965 on 2022-06-07:
Something you could check to see if it's about your code. Try and run 5-6 default nodes. Basically run a couple of the tutorial code at the same time. If that causes the same problem it's not your code.

Comment by alberto on 2022-06-07:
Thank you @Joe29965 , I've tried running the simple talker and listener of the tutorial. Sadly I can reproduce the same exact errors :(

I have also reinstalled ROS2 and created a new ws (still keeping the old one). Sourcing and building with the new setup.bat, but I still face the bug with the tutorial code.

Comment by Joe28965 on 2022-06-07:
Honestly, I do not know what it might be. I have only ever used Ubuntu to run ROS.

Comment by gvdhoorn on 2022-06-08:
Could you please stop bumping your message with trivial edits?

Comment by alberto on 2022-06-08:
Yes sorry, I wanted to find a more suitable title :P. Anyways, using WinDgb and scanning the working and not working nodes, I found out some differences. In particular the node that hangs doesn't "load" these .dll:

ModLoad: .. C:\dev_new\ros2-windows\bin\rmw_dds_common__rosidl_typesupport_fastrtps_cpp.dll ModLoad: .. C:\dev_new\ros2-windows\bin\rcl_interfaces__rosidl_typesupport_fastrtps_c.dll
ModLoad: .. C:\dev_new\ros2-windows\bin\builtin_interfaces__rosidl_typesupport_fastrtps_c.dll
ModLoad: .. C:\dev_new\ros2-windows\bin\rcl_interfaces__rosidl_typesupport_fastrtps_cpp.dll
ModLoad: .. C:\dev_new\ros2-windows\bin\builtin_interfaces__rosidl_typesupport_fastrtps_cpp.dll
ModLoad: .. C:\dev_new\ros2-windows\bin\std_msgs__rosidl_typesupport_fastrtps_cpp.dll

But obviously the cmd doesn't return any errors. Can this cause my bug?

Comment by gvdhoorn on 2022-06-09:
No idea. We also don't yet know whether this is "a bug".

You mentioned you've "reinstalled ROS2": how exactly?

And did you also clean out your workspace and rebuild it completely after reinstalling ROS?

Comment by alberto on 2022-06-09:
I wanted to keep the old underlay and overlay, so to reinstall ros2 I did this: I went here, I downloaded again ros2-foxy-20220208-windows-release-amd64.zip and extracted it in a new folder (underlay). Then I created a new dev_ws (overlay), I put cpp_pubsub in the src and build it. But this didn't change my problem. So I tried to re-download the other components (but not removing the old ones). I re-download Python, VC++, OpenSLL and catkin_pkg, cryptography etc using the command suggested in the link (I didn't re-download choco). Obviously they were already installed, so I think I didn't change anything, in fact the cmd always said something like: "Already up to date".

Comment by alberto on 2022-06-09:
Adding some info. I have used ros2 run cpp_pubsub listener --ros-args --log-level debug and: On not working node I only get these 3 lines:

-[DEBUG] [1654776889.921857300] [rclcpp]: signal handler installed

-[DEBUG] [1654776889.922523400] [rcl]: Initializing node 'minimal_subscriber' in namespace ''

-[DEBUG] [1654776889.922739800] [rcl]: Using domain ID of '0'

On a working node, the next lines should be: -[DEBUG] [1654776963.071939700] [rcl]: Initializing publisher for topic name '/rosout'

-[DEBUG] [1654776963.072044000] [rcl]: Expanded topic name '/rosout'

-[DEBUG] [1654776963.074039400] [rcl]: Publisher initialized

-[DEBUG] [1654776963.074124300] [rcl]: Node initialized

etc.

Comment by genevanmeter on 2022-06-17:
I am also experiencing this issue on Windows 10 with Humble. It's like there is a limit to how many simultaneous connections or possibly Windows Firewall or Defender's real-time protection kicking in. Not having this issue on Ubuntu.

Answer 1

We observed a similar issue and I found setting up specific/custom transport protocols seemed to fix it for us.

I set up a UDP and TCP transport interface (omitting shared memory transport) which may suggest that the collision/hang occurs in setting up the shared memory space which I assume is probably enabled by default.

So create this DDS profile (fastdds_profiles.xml)

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
    <transport_descriptors>
        <transport_descriptor>
            <transport_id>CustomTcpTransport</transport_id>
            <type>TCPv4</type>
            <interfaceWhiteList>
                <address>127.0.0.1</address>
            </interfaceWhiteList>
        </transport_descriptor>
        <transport_descriptor>
            <transport_id>CustomUdpTransport</transport_id>
            <type>UDPv4</type>
            <interfaceWhiteList>
                <address>127.0.0.1</address>
            </interfaceWhiteList>
            <non_blocking_send>false</non_blocking_send>
        </transport_descriptor>
    </transport_descriptors>

    <participant profile_name="CustomTransportParticipant" is_default_profile="true">
        <rtps>
            <useBuiltinTransports>false</useBuiltinTransports>
            <userTransports>
                <transport_id>CustomTcpTransport</transport_id>
                <transport_id>CustomUdpTransport</transport_id>
            </userTransports>
        </rtps>
    </participant>
</profiles>

and then make it available before launching nodes

set FASTRTPS_DEFAULT_PROFILES_FILE=fastdds_profiles.xml

Note: this profile also limits discoverability to loopback/localhost (127.0.0.1) more network adapters can be added)

Background: Even with the hanging nodes I observed that the ports for all nodes were created which may suggest why they can still be observed from a linux machine, but the nodes didn't completely start which would suggest the nodes are likely to still be non-functional.

Answer 2

---> LAST EDIT: solution<---

I've found that changing the RMW implementation does the work. I was using the default one (rmw_fastrtps_cpp), and I changed it to rmw_cyclonedds_cpp. Now, without reinstalling anything, ROS seems to work normally.

Temporary solution:

I decided to reset my PC and after I installed all the things, it seems the problem is gone. I guess something gets corrupted between fastrtps and Windows. Anyways, using the same code with rmw_fastrtps_cpp will lead you to the same situation as before.

---

Originally posted by alberto with karma: 100 on 2022-06-13

This answer was ACCEPTED on the original site

Post score: 0

---

Original comments

Comment by genevanmeter on 2022-06-18:
My comment was hidden under see more comments but I am also experiencing this Windows. I'll add that when experiencing the issue on Windows I switched over to a separate machine running Ubuntu and was able to run ros2 topic list, subscribe, etc for nodes running on that Windows machine. So the issue was occurring only on the "host" Windows machine but external connections were fine.

I don't know where to begin to diagnose it.

Comment by alberto on 2022-06-18:
Hi, may I ask you what type of code operation are you doing? Are you using dynamic allocation like me? Are you sending sensor_msgs.Image[n] ? Because from what I've experienced, seems that there's some code (probably bad written by me) that causes all the problem, and then affects all the ros system. In fact using a different PC or a making fresh windows installation (note that only fresh ros2 installation doesn't resolve the problem), I don't have any problems running even 8 nodes simultaneously of the cpp_pubsub tutorial. However, I managed to corrupt again the "fresh windows installation" with the code above. My guess is that ROS2 on Windows miss some preventions for this type of situations.

Comment by genevanmeter on 2022-06-19:
New to this but as I understand it yes. I am creating/destroying topics during runtime via service calls. The service call takes an int which is appended to the topic name. The msg format also contains a variable length array which is also dynamic so the total size of the msg is changing within any given topic.

Comment by buckley.toby on 2023-07-13:
I can't find clear instructions on installing cyclonedds on windows, can someone advise?

Answer 3

I also have the same issue.I am using ros2 humble in windows10(virtual box).Ros2 commands are becoming unresponsive ..Even rviz2 is not working.I have posted the rviz2 issue in my page.