0
$\begingroup$

Rosanswers logo

I'm new to ROS2. I've got a somewhat weird setup that I'm trying to launch. My issue is that minimal publisher and subscriber from examples don't hear each other.

I've got a docker container, based on Ubuntu 18.04, with ROS2 Foxy installed from sources. This container has installed environment that I have to use, namely, CUDA 10.0 and some software using it.

This container is run on Ubuntu 20.04 host. This host has also installed ROS2 Foxy from Debian packages.

I run the container with --net host, then launch minimal subscriber from examples in it. Then I run minimal publisher on the host.

That is on host:

$ docker run --rm -it --net host container

In container:

# ros2 run examples_rclpy_minimal_subscriber subscriber_member_function

On host in another terminal:

$ source /opt/ros/foxy/setup.zsh
$ ros2 run examples_rclpy_minimal_publisher publisher_local_function

The issue is that the subscriber doesn't hear anything from the publisher. The same issue is observed, if I swap subscriber and publisher, that is run publisher in the container and subscriber on the host.

If I execute both subscriber and publisher in the container, subscriber hears it. On host:

docker exec -it container /bin/bash

In container:

ros2 run examples_rclpy_minimal_publisher publisher_local_function
    

And subscriber prints "I heard hello".

At the same time, multicast messages (ros2 multicast send, ros2 multicast receive) are sent and received successfully.

If I run the container without --net key, then the subscriber running in it, sees messages from the publisher. However, this limits usage of the container. It won't be seen from another PC.

I've pasted the minimal Dockerfile illustrating the issue at the bottom of this message (unfortunately, I cannot attach it).

I've also pushed image to the dockerhub, and it can be downloaded with the command docker pull wl2776/nvidia-ros:latest

What I've also checked.

  1. Running netcat both in container and on host shows that the traffic is passed through the container boundary. If I replace subsriber in the container with netcat it prints several packets. That is, on host:

     $ docker run --rm -it --net host container
        $ docker cp /bin/nc.openbsd container:/tmp
    

    In container:

     # /tmp/nc.openbsd -l -u -p 7400 | od -
    

    On host in another terminal:

     $ source /opt/ros/foxy/setup.zsh
        $ ros2 run examples_rclpy_minimal_publisher publisher_local_function
    

    And I see several dumps from nc

  2. I've also tried the experiment with minimal publisher and subscriber using docker image tagged osrf/ros:foxy-desktop from dockerhub. Everything has worked perfectly.

  3. When minimal subscriber runs in the container, its node and topic are seen on the host. On host, while subscriber in the container is running:

     $ ros2 topic list
        /parameter_events
        /rosout
        /topic
        $ ros2 node list
     /minimal_subscriber
    

    If I stop minimal subsriber in the container with Ctrl-C, then it disappears from ros2 node list

  4. Sniffing packets with tcpdump shows that subscriber in OSRF's container sends RTPS packets with version 2.3, but mine - with version 2.2. They also have different sizes (252 bytes from OSRF's container and 308 bytes from mine). Could these differences be the reason of the issue?

  5. ros2 doctor --report prints that RMW middleware is "fastrtps_cpp", both in container and on host. It also prints 3 network interfaces (docker0, ethernet and loopback) both on host and in container. docker and ethernet interfaces have MULTICAST enabled.

  6. If I run in parrallel my container and the container, published by OSRF, then subscriber in the latter hears "Hello's" from the publisher in the former. On host: docker run --rm -it --net host wl2776/nvidia-ros:latest, in the container: ros2 run examples_rclpy_minimal_publisher publisher_local_function

    On host in another terminal: docker run --rm -it --net host osrg/ros:foxy-desktop, in the container: export ROS_DOMAIN_ID=142 && ros2 run examples_rclpy_minimal_subscriber subscriber_member_function

The Dockerfile

FROM nvidia/cuda:10.0-devel-ubuntu18.04

ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update \
 && apt-get install -y --no-install-recommends locales curl gnupg2 lsb-release apt-transport-https ca-certificates gnupg software-properties-common \
 && rm -rf /var/lib/apt/lists/* 

RUN locale-gen en_US en_US.UTF-8 && update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
ENV LANG=en_US.UTF-8

RUN curl -s https://raw.githubusercontent.com/ros/rosdistro/master/ros.asc | apt-key add -
RUN echo "deb [arch=$(dpkg --print-architecture)] http://packages.ros.org/ros2/ubuntu $(lsb_release -cs) main" > /etc/apt/sources.list.d/ros2-latest.list

RUN apt-add-repository ppa:lttng/stable-2.11 

# ROS2 build uses `target_library_directories` that appeared in 3.13
# Original image already contains CMake, but its version is too old, 3.10.
# Therefore install more recent cmake.
# Latest version, 3.19, appears too noisy, it outputs many deprecation warnings, 
# therefore we install 3.18
RUN curl -s https://apt.kitware.com/keys/kitware-archive-latest.asc | gpg --dearmor - > /etc/apt/trusted.gpg.d/kitware.gpg 

RUN apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main' \
 && apt-get update \
 && apt-get install -y --no-install-recommends cmake-data=3.18.4-0kitware1 cmake=3.18.4-0kitware1 

RUN apt-get install -y --no-install-recommends \
 adwaita-icon-theme build-essential clang-format clang-format-6.0 clang-tidy \
 clang-tidy-6.0 cppcheck dconf-gsettings-backend dconf-service dh-python \
 fonts-liberation gazebo9 git google-mock googletest graphviz libann0 libapr1 \
 libapr1-dev libaprutil1 libaprutil1-dev libasio-dev libassimp-dev libassimp4 \
 libbenchmark-dev libboost-dev libboost-filesystem-dev libboost-test-dev \
 libboost-thread-dev libbullet-dev libcdt5 libcgraph6 libconsole-bridge-dev \
 libcppunit-1.14-0 libcppunit-dev libcunit1-dev libcurl4-openssl-dev \
 libeigen3-dev libexpat1-dev libfreetype6-dev libgazebo9-dev libgl1-mesa-dev \
 libglfw3-dev libglu1-mesa-dev libgtest-dev libgtk-3-dev libgts-0.7-5 libgts-bin \
 libice-dev libldap2-dev liblog4cxx-dev liblog4cxx10v5 liblttng-ctl0 \
 liblttng-ust-dev liboctomap-dev libopencv-calib3d-dev libopencv-core-dev \
 libopencv-dev libopencv-features2d-dev libopencv-flann-dev \
 libopencv-imgcodecs-dev libopencv-imgproc-dev libopencv-ml-dev \
 libopencv-photo-dev libopencv-shape-dev libopencv-superres-dev \
 libopencv-video-dev libopencv-videoio-dev libopencv-videostab-dev \
 libopencv-viz-dev libpcl-* libpython3-dev libpython3.6 libpython3.6-dev \
 libqt5designer5 libqt5help5 libqt5opengl5 libqt5svg5 libsm-dev libspdlog-dev \
 libsqlite3-dev libssl-dev libtinyxml-dev libtinyxml2-dev liburcu6 \
 libusb-1.0-0-dev libxaw7-dev libxml2-utils libxmu-dev libxmu-headers libxpm-dev \
 libxrandr-dev libxrender-dev libxslt1.1 libxt-dev libyaml-dev libzstd-dev \
 lttng-modules-dkms lttng-tools ntp openssh-client pkg-config pydocstyle pyflakes pyflakes3 \
 pyqt5-dev python-pkg-resources python-pyflakes python3-asn1crypto \
 python3-babeltrace python3-bs4 python3-cffi-backend python3-chardet \
 python3-colcon-common-extensions python3-cryptography python3-dev \
 python3-flake8 python3-html5lib python3-idna python3-ifcfg python3-lark-parser \
 python3-lttng python3-lxml python3-matplotlib python3-mock python3-mypy \
 python3-netifaces python3-nose python3-numpy python3-pip python3-psutil \
 python3-pydocstyle python3-pydot python3-pygments python3-pygraphviz \
 python3-pyqt5 python3-pyqt5.qtsvg python3-pytest-cov python3-pytest-mock \
 python3-rosdep python3-setuptools python3-sip python3-sip-dev \
 python3-snowballstemmer python3-vcstool python3-webencodings qtbase5-dev \
 sip-dev uncrustify x11proto-randr-dev \
  && rm -rf /var/lib/apt/lists/*

RUN pip3 install --upgrade --no-cache-dir pip
RUN pip3 install -U --no-cache-dir argcomplete wheel
RUN pip3 install -U --no-cache-dir flake8-blind-except flake8-builtins flake8-class-newline flake8-comprehensions flake8-deprecated \
 flake8-docstrings flake8-import-order flake8-quotes pytest-repeat pytest-rerunfailures pytest \
 tensorflow==1.14.0 numpy Pillow opencv-python pycuda \
 && rm -rf /tmp/*

WORKDIR /root
RUN mkdir -p ~/ros2_foxy/src && cd ~/ros2_foxy && curl https://raw.githubusercontent.com/ros2/ros2/foxy/ros2.repos | vcs import src

WORKDIR /root/ros2_foxy
RUN rosdep init && rosdep update && \
 rosdep install --from-paths src --ignore-src --rosdistro foxy -y --skip-keys "console_bridge fastcdr fastrtps rti-connext-dds-5.3.1 urdfdom_headers" \
 && rm -rf /var/lib/apt/lists/* 

WORKDIR /root
RUN colcon build --merge-install --install-base=/opt/ros/foxy \
 && rm -rf /var/lib/apt/lists/* /root/build /root/.ros /root/log

Originally posted by wl on ROS Answers with karma: 127 on 2021-01-29

Post score: 3


Original comments

Comment by jacobperron on 2021-02-04:
I tried to reproduce on top of an 18.04 docker container, but the examples work for me (publishing from host and subscribing from inside the Docker container). Here's the Dockerfile I used.

Comment by wl on 2021-02-05:
I've built an image from your Dockerfile. Still the same issue. Did you use --net host? It is important.

Everything works without --net host, but fails, if network isolation is turned off.

Comment by mschratter on 2021-02-11:
Did you find a solution or could you isolate the problem? I have the same issue. Without --net host the communication works, but of course only isolated and not together with another computer in the network.

Comment by wl on 2021-02-11:
I think, the issue is in FastRTPS. If at least one of nodes (host or containerized, no matter) switches to CycloneDDS, the "hello" messages become heard. Here is the link to discussion on Github: https://github.com/eProsima/Fast-DDS/issues/1698 They have deleted my comment that everything works, if I switch to CycloneDDS.

$\endgroup$

4 Answers 4

0
$\begingroup$

Rosanswers logo

The last releases of Fast-DDS come with SharedMemory transport by default. Using --net=host implies both DDS participants believe they are in the same machine and they try to communicate using SharedMemory instead of UDP. Fast-DDS team will work in to implement a mechanism to detect this kind of situation. Meanwhile, I can give you two solutions:

  • Using a XML to disable SharedMemory transport in one of the DDS participants.

XML:

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles" >
    <transport_descriptors>
        <transport_descriptor>
            <transport_id>CustomUdpTransport</transport_id>
            <type>UDPv4</type>
        </transport_descriptor>
    </transport_descriptors>

    <participant profile_name="participant_profile" is_default_profile="true">
        <rtps>
            <userTransports>
                <transport_id>CustomUdpTransport</transport_id>
            </userTransports>

            <useBuiltinTransports>false</useBuiltinTransports>
        </rtps>
    </participant>
</profiles>
  • Enable SharedMemory between host and container. For this you should share /dev/shm:
    docker run -ti --net host -v /dev/shm:/dev/shm <DOCKER_IMAGE>
    Also, both applications should be run with the same UID. In my case, my docker container's user is root (UID=0). Then I had to run the host application as root.

I hope this helps you.


Originally posted by richiware with karma: 106 on 2021-02-12

This answer was ACCEPTED on the original site

Post score: 9


Original comments

Comment by BhanuKiran.Chaluvadi on 2022-01-06:
@richiware. So by using XML way can we avoid using --net=host . Because using host network for containers will compromise lot of security.

Comment by Markus Bader on 2022-09-20:
I had the same issue between two docker containers. Mounting the shared memory on both with -v /dev/shm:/dev/shm solved the problem. Thanks

$\endgroup$
1
$\begingroup$

I have also encountered this issue and haven't been able to come up with a fix, but some workaround. The error seems to be rooted in Fast-DDS trying to use SHM when using the --net=host option. The issue has also been described multiple times, e.g. https://github.com/eProsima/Fast-DDS/issues/3475, https://github.com/eProsima/Fast-DDS/issues/2956, https://stackoverflow.com/questions/65900201/troubles-communicating-with-ros2-node-in-docker-container.

My workaround to use SHM between ROS 2 nodes inside containers and on the host is to:

  1. start the container with these options: --network=host --ipc=host --pid=host
  2. run nodes inside the container with the same user id as nodes on the host

For convenience, I have created a repository, https://github.com/rosblox/ros-template, to build (as root user) and run (with host's user id) ROS 2 packages/ nodes inside containers. This allows to use Shared Memory Transport (SHM) between ROS 2 nodes run inside the container and on the host.

$\endgroup$
1
$\begingroup$

I had a problem with some of the same symptoms as this and spent quite a long time looking at all the DDS related configuration, but eventually figured out that it was because I set up rootless mode on my docker engine (as prompted to do so during docker installation) instead of just adding myself to the docker group.

Not a direct solution to the above problem, but it may help someone.

$\endgroup$
0
$\begingroup$

Rosanswers logo

As described here, "the last releases of Fast-DDS come with SharedMemory transport by default. Using --net=host implies both DDS participants believe they are in the same machine and they try to communicate using SharedMemory instead of UDP".

Can be solved with additional XML config, disabling shared memory:

  1. $ cat > fastrtps-profile.xml and paste XML from the link above
  2. $ export FASTRTPS_DEFAULT_PROFILES_FILE=$(pwd)/fastrtps-profile.xml

Originally posted by wl with karma: 127 on 2021-02-12

This answer was NOT ACCEPTED on the original site

Post score: 3

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.