Optimization of the SingleThreadedExecutor and MultiThreadedExecutor to reduce CPU overhead.

Feature request / Bug report

Optimization of the SingleThreadedExecutor and MultiThreadedExecutor to reduce CPU overhead.
Possible solutions:

• Overhaul design of current implementation
  • Work with callback groups instead of nodes
  • Use events to notify executor of changes to system instead of constant checking with weak nodes

OR

• Add a separate optimized Executor that is less dynamic (more user configuration)

Feature description

We did research on the large CPU overhead introduced by the Executor in rclcpp and found it to be a blocking issue for our ROS 2 platform. The results of the research can be found here: https://github.com/nobleo/ros2_performance . As a result of the research a discussion was started here:
https://discourse.ros.org/t/singlethreadedexecutor-creates-a-high-cpu-overhead-in-ros-2/10077 . People who joined the discussion also looked into the CPU overhead and came to the same conclusions. In our opinion changes to the Executor or alternatives to the current existing Executor are necessary for normal ROS 2 to run on embedded boards.

We are continuing our research by implementing a custom static Executor as a POC to get an idea of the maximum CPU gain that is possible. Code will be made available when we have something functional and the results of the research will be shared on our github.

Additionally the current implementation of the Executor has some weird effects that are discussed here https://t-blass.de/papers/response-time-analysis-of-ros2.pdf . The paper focuses more on how to measure response time than on the nature of the Executor. However, the paper does highlight the Executors peculiar preference for timers and the weird priority that is based on CallbackType and RegistrationTime.

The improvements that could be made to scheduling (that are required for real-time) and the optimization with respect to CPU overhead could/should both be encapsulated in a new and improved Executor.

Related issues

These issues need to be considered when redesigning the executor.
#392
#519
#726

Edit:
The author of the paper mentioned above raised this issue:
#633

Implementation considerations

Major changes to the Executor in rclcpp (and possibly to the node class and others) could break existing user code. Implementing changes will take a lot of effort and may also involve changes in other abstraction layers of ROS2.

Start by reading the current SingleThreadedExecutor and MultiThreadedExecutor implementations in rclcpp, then review the linked performance research and discussion. Compare the existing executor behavior with the proposed callback-group and event-driven designs. Done would require a defined executor redesign or alternative with reduced CPU overhead and considered scheduling behavior, while preserving user compatibility.