Executor callbacks are no longer in a predictable order

Bug report

Required Info:

• Operating System:
  • Ubuntu 20.04
• Installation type:
  • Source
• Version or commit hash:
  • https://github.com/ros2/ros2/blob/d4c09985560a2357ad7fca9e20f39d9ab040666b/ros2.repos
• DDS implementation:
  • Fast-RTPS
• Client library (if applicable):
  • rclcpp

Steps to reproduce issue

1. Build the https://github.com/boschresearch/ros2_response_time_analysis project against ROS2 Jazzy (note that some minimal changes are required).
2. Observe that the results of running the test show that the callbacks are executed in an arbitrary order as compared to the previous ordering which was with respect to the order in which the subscriptions/services were added to the node.

Expected behavior

With ROS2 Eloquent through Iron, the callbacks all executed in the following order, where the ordering was defined by the order in which the subscriptions/services were added to the node.

Actual behavior

With ROS2 Jazzy, the ordering is now arbitrary and not related to the order in which messages were received or the order in which subscriptions were added to the node. The execution order of timers also appears to be affected.

NOTE: While the low, medium, and high subscriptions are processed in the same order that they are transmitted, I believe this to be a fluke as the service and timer callbacks are not processed in the order they were registered or sent. For reference, the callbacks are registered in the following order:

1. high
2. medium
3. low
4. srv_high
5. srv_medium
6. srv_low
7. timers (as directed by the test)

And the topics/services are sent in the following order:

1. low
2. medium
3. high
4. srv_high
5. srv_low
6. low
7. medium
8. high
9. srv_high
10. srv_low
11. srv_medium
12. srv_medium
13. high

With timers set to expire at 200, 200, 2300, and 2300 milliseconds into the test.

Additional information

I believe this change is a result of #2142 due to the EntityCollection class being based on std::unordered_map where the key is the handle returned by rcl for the timer/subscription/etc. The entities are then added to the wait set in whatever order they are iterated through by the std::unordered_map and thus processed in that order. I haven't had the opportunity to test it, but I believe the ordering could be different from system to system or even run to run as the rcl handle values change.

The default AllocatorMemoryStrategy used in previous versions of rclcpp utilized std:vector for storing rcl handles, which maintained the order in which they were added to the node.