Status of today's ROS 2 message delivery mechanisms

Opening a new ticket to follow up on some of the discussions that are going on in the ROS 2 Middleware WG and in https://github.com/ros2/rclcpp/issues/1642

As of today, ROS 2 supports various mechanisms to deliver messages between nodes, depending on where they are located and on various optimization configurations.

Intra Process communication

This mechanism allows to pass smart pointers between nodes in the same process.
By being developed in rclcpp, the RMW is not anymore the only entity responsible for passing messages around. This may be considered a philosophical/design problem, but it also has very practical implications.

PROS:

• It has the best performance in terms of CPU and latency, small RAM overhead.
• It's independent from the chosen RMW.

CONS:

• Maintenance cost: need to re-implement pub/sub/services/clients/actions as waitables rclcpp entities and QoS settings.
• Huge overhead when you have both inter and intra process readers: https://github.com/ros2/rclcpp/issues/1722

Loaned message

This mechanism asks the RMW to create a message (which can be done from a shared memory segment).
Then this message can be efficiently passed around across multiple processes in the same machine.
See more details about performance in https://github.com/ros2/rclcpp/issues/1642#issuecomment-900387789

PROS:

• Good performance for big messages

CONS:

• Uses a separate set of APIs so a node must know in advance if it will need to use loaned messages.
• Bad performance for small messages passed in a single process (equivalent to run without any optimization).
• No support for dynamic data-types.

Standard RMW delivery

This is the standard mechanism that RMW uses to deliver messages regardless of the location of the entities.

PROS:

• Common API

CONS:

• Bad performance for single process and single machine communication.

The purpose of this ticket is to have a discussion on how to improve these mechanisms and have them coexist better.

FYI @wjwwood @mauropasse @ivanpauno @fujitatomoya

Start by reviewing the message delivery mechanisms described here and the linked rclcpp issue 1642, then read the related Middleware WG discussion. No file or test is named. This ticket is complete only when the project agrees on how intra-process, loaned-message, and standard RMW delivery should coexist.