Debugging ROS2 rclpy Service Call

Question

I have a project using ROS2 Humble running on Ubuntu 22.04. I’m looking for advice on debugging a issue with a synchronous service call.

middleware: rmw_fastrtps_cpp

Context

I’ve completed a large refactor as well as some new development which included changes to both the node that host the service client and the one that host the service server. However, the service itself and the callback internals haven’t changed. Snippet of code of the service setup and use Node 1 - Service Server

self.create_service(srv_type=MapLocation,
                            srv_name='/exploration_goal',
                            callback=self.get_exploration_goal)

Node 2: Service Client

self._srv_client = self.node.create_client(
            srv_type=MapLocation,
            srv_name='/exploration_goal',
            callback_group=MutuallyExclusiveCallbackGroup())

Node 2: Service call

    if not self._srv_client.service_is_ready():
        raise RuntimeError(
            '/exploration_goal Service unresponsive')

    call_start = self.node.get_clock().now()
    result: MapLocation.Response = self._srv_client.call(request)

Issue

At the start of a new run the service works as expect, executing from call to result within <1sec. However, after a while the service start to hang taking up to minutes to return a result. It seem like the service get slower and slower with each call. After some investigation, I’ve observed that the delay occurs between the service call and the callback execution. The callback execution and return of the result itself run in the expect time. I’ve tried:

1. Cleaning and rebuilding all packages
2. Running in a docker (From ros:humble-ros-base-jammy) I would greatly appreciate any guidance on debugging such use.
3. Running on different machine