ROS2 connectivity across Docker containers via Host Driver

Question

I'm having some issues connecting ROS2 nodes across Docker containers via the host driver. Reading through some relevant RTI documentation: How to use RTI Connext DDS to Communicate Across Docker Containers Using the Host Driver

I've concluded my main issue stems from this:

As we mentioned before, the host Docker driver mode could affect this generation process as follows:

All the containers in host driver mode will have the same IP address; therefore, they will have the same Host ID.

Due to the way process IDs are generated, two applications in two containers could have the same Process ID; therefore, they could have the same Application ID. In this case, RTI Connext DDS will interpret that both applications are in the same machine and will try to communicate using shared memory.

Suggested workarounds include; enabling communication using shared memory among container by passing the --ipc=host argument: Communicate two Docker containers using RTI Connext DDS and shared memory.

Among the rwm implementations I've tested, only rmw_opensplice_cpp seems to be working, however I suspect rtps_auto_id_kindhas been set to RTPS_AUTO_ID_FROM_UUID for opensplice, given the participant GUIDs seems randomized; not confirmed though as I didn't look at it too closely. Controlling How the GUID is Set (rtps_auto_id_kind)

Here is a minimal example to demonstrate the issue. You'll notice that the listener will print the chatter messages from the talker if neither docker-compose service is set to use the host network driver, or only one is, but not such when both are. Even then, enabling IPC does not seem to resolve anything here.

mkdir ros2_ipc && cd ros2_ipc
touch Dockerfile
touch docker-compose.yml
docker-compose up

Dockerfile:

FROM ros:bouncy

ENV RTI_NC_LICENSE_ACCEPTED yes

RUN apt-get update && apt-get install -y \
      ros-$ROS2_DISTRO-rmw-connext-cpp \
      ros-$ROS2_DISTRO-rmw-opensplice-cpp && \
    rm -rf /var/lib/apt/lists/*

docker-compose.yml

version: '3'

services:
  talker:
    build: .
    environment:
      - "ROS_DOMAIN_ID=0"
      # - "RMW_IMPLEMENTATION=rmw_connext_cpp"
      # - "RMW_IMPLEMENTATION=rmw_fastrtps_cpp"
      # - "RMW_IMPLEMENTATION=rmw_opensplice_cpp"
    # network_mode: "host"
    # ipc: host
    command: ros2 run demo_nodes_cpp talker

  listener:
    build: .
    environment:
      - "ROS_DOMAIN_ID=0"
      # - "RMW_IMPLEMENTATION=rmw_connext_cpp"
      # - "RMW_IMPLEMENTATION=rmw_fastrtps_cpp"
      # - "RMW_IMPLEMENTATION=rmw_opensplice_cpp"
    # network_mode: "host"
    # ipc: host
    command: ros2 run demo_nodes_cpp listener

TL;DR Anyone know how to easily set a different RTPS App ID for each DDS participant to uniquely identify them across containers, or if I can set rtps_host_id QoS parameter via ROS2 while remaining vendor agnostic?

---

Originally posted by ruffsl on ROS Answers with karma: 1094 on 2018-07-10

Post score: 5

---

Original comments

Comment by ruffsl on 2018-07-11:
It has come to my attention that the use of host driver for both nodes works otherwise when security is enabled for fastrtps and connext: https://github.com/ruffsl/ros2_docker_demos/issues/1

Comment by jadran on 2020-03-27:
Docker network architecture covers all network scenarios required for successful communication at the local, remote or cluster level.

Answer 1

The issue seems to be related to Fast-DDS trying to establish connections over SHM. The issue has been described multiple times, latest here: https://github.com/eProsima/Fast-DDS/issues/3475

I haven't found a solution to it, but a workaround. The workaround is to:

1. start the container with these options: --network=host --ipc=host --pid=host
2. run nodes inside the container with the same user id as nodes on the host

For convenience, I have created this repository, https://github.com/rosblox/ros-template, to build (as root user) and run (with host's user id) ROS 2 packages/ nodes inside containers. This allows to use Shared Memory Transport (SHM) between ROS 2 nodes run inside the container and on the host.

Answer 2

Well, I found some workarounds.

PID settings (--pid)

One way to ensure the Application IDs (ergo GUIDs) for the two DDS participants in separate containers using the host's network driver do not collide is to merely have the containers also share a common PID namespace; thus allocating unique PIDs the process of the DDS participants.

You can either set the PID (Process) Namespace mode for the container to 'container:<name|id> to joins another container's PID namespace, or host to use the host's PID namespace inside the container. The later sort of defeats any namespace isolation, and the former can result in some unintended behaviors when restarting containers. Additionally, when using compose, race conditions may surface when docker-compose attempts to initiate a container with a PID namespace of another container that has itself not been started.

https://docs.docker.com/engine/reference/run/#pid-settings---pid
https://docs.docker.com/compose/compose-file/#pid

Here is an example where an third intermediary container is used to host the PID namespace (and sharable IPC set for Inter-Process communication), yet compose will error with the message on how the container service intermediary is not available. Attempting to resolve the use by declaring the bootstrapping dependency via the depends_on setting does not resolve this, so one must manually create such a container (say by running docker-compose up with only the intermediary service uncommitted out, then repeating up with the rest uncommented).

version: '3'

services:
  intermediary:
    build: .
    ipc: shareable
    command: sleep infinity

  talker:
    build: .
    environment:
      - "ROS_DOMAIN_ID=0"
      - "RMW_IMPLEMENTATION=rmw_connext_cpp"
      # - "RMW_IMPLEMENTATION=rmw_fastrtps_cpp"
      # - "RMW_IMPLEMENTATION=rmw_opensplice_cpp"
    network_mode: "host"
    ipc: container:demo_intermediary_1
    pid: container:demo_intermediary_1
    command: ros2 run demo_nodes_cpp talker
    depends_on:
      - "intermediary"

  listener:
    build: .
    environment:
      - "ROS_DOMAIN_ID=0"
      - "RMW_IMPLEMENTATION=rmw_connext_cpp"
      # - "RMW_IMPLEMENTATION=rmw_fastrtps_cpp"
      # - "RMW_IMPLEMENTATION=rmw_opensplice_cpp"
    network_mode: "host"
    ipc: container:demo_intermediary_1
    pid: container:demo_intermediary_1
    command: ros2 run demo_nodes_cpp listener
    depends_on:
      - "intermediary"

It would seem this is a bit cumbersome when using docker-compose. As mentioned setting pid to host simplifies a few of these race condition issues, but lose some level of isolation that may be preferable. A better approach would be configure the Application ID to be unique through QoS settings, though there doesn't seem to be a simple way to achieve this for ROS2 as of writing.

---

Originally posted by ruffsl with karma: 1094 on 2018-07-23

This answer was ACCEPTED on the original site

Post score: 2