Inter-process communication causes increased latency of intra-process communication

Bug report

Required Info:

• Operating System:
  • Ubuntu 20.04
• Installation type:
  • Both binaries and from source
• Version or commit hash:
  • Galactic (binaries) and master dbb717cd6ee838be4cb9b2f72b1804d071c6515f (from source)
• DDS implementation:
  • Cyclone DDS
• Client library (if applicable):
  • rclcpp

Steps to reproduce issue

1. Follow the instructions from https://github.com/mitsudome-r/pointcloud_delay_check. It launches a container with two nodes, publisher and subscriber of a pointcloud, that use intra-process communication.
2. Create inter-process communication, e.g. ros2 topic hz -w 10 /pointcloud

Expected behavior

Intra-process communication latency is not affected by inter-process communication.

Actual behavior

Intra-process communication latency is heavily increased by the existence inter-process communication.

Additional information

I profiled the container:

• with only intra-process communication (right)
• when additional inter-process communication was established via ros2 topic hz -w 10 /pointcloud (left)

The profiling indicates that the delay is caused by rclcpp::Executor::execute_subscription that is called only when inter-process communication is established. It seems like a bug.

I tried out these modifications which calls add_subscription only for inter-process. Therefore, execute_subscription is not called for this process.

diff --git a/rclcpp/src/rclcpp/node_interfaces/node_topics.cpp b/rclcpp/src/rclcpp/node_interfaces/node_topics.cpp
index c57fbcee..a72d5b5d 100644
--- a/rclcpp/src/rclcpp/node_interfaces/node_topics.cpp
+++ b/rclcpp/src/rclcpp/node_interfaces/node_topics.cpp
@@ -95,17 +95,17 @@ NodeTopics::add_subscription(
     callback_group = node_base_->get_default_callback_group();
   }
 
-  callback_group->add_subscription(subscription);
-
-  for (auto & subscription_event : subscription->get_event_handlers()) {
-    callback_group->add_waitable(subscription_event);
-  }
-
   auto intra_process_waitable = subscription->get_intra_process_waitable();
   if (nullptr != intra_process_waitable) {
     // Add to the callback group to be notified about intra-process msgs.
     callback_group->add_waitable(intra_process_waitable);
-  }
+  } else {
+    callback_group->add_subscription(subscription);
+
+    for (auto & subscription_event : subscription->get_event_handlers()) {
+      callback_group->add_waitable(subscription_event);
+    }
+  };
 
   // Notify the executor that a new subscription was created using the parent Node.
   {

Communication seems to work properly as both subscribers receive the data. Below there is a plot comparing original behavior (orange) and the changed (blue) one. The measured value is the delay for intra-process communication. The inter-process communication is switched on and off periodically. The modification reduces the delay significantly.