use_intra_process_comms does not work with LifecycleNode transitions for BondCpp
@mjcarroll is already working on this.
Since Jan 30, 2025.
- Dominant language
- C++
- Stars
- 805
- Forks
- 564
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 27
Description
Bug report
Required Info:
- Operating System: Ubuntu 22.04 (using OSRF Ubuntu 24.04 docker)
- Installation type: Binaries
- Version or commit hash: Latest Rolling 29.3.0
- DDS implementation: Fast-DDS
- Client library (if applicable): rclcpp
Expected behavior
IPC enabled has the system able to function correctly with bond, lifecycle nodes.
Actual behavior
IPC enabled does not work with lifecycle and bond nodes. Servers crash when destroying bond connections in on_deactivate lifecycle transitions with intra-process related traces.
Additional information
I'm working on migrating Nav2 over to using IPC and ran into another issue after #2704 and #2705. I now generally have data flowing around, but we fail to be able to cleanly shutdown any lifecycle nodes when IPC is enabled, both when composed into the same container and also when the nodes are in separate processes. This causes all of our system-level tests to fail, as well as a couple of unit tests that are nice for reproducibility. We have never seen this issue before or without using IPC, so I'm reasonably convinced now its an issue with the IPC manager on shutdown related tasks.
I have backtraces and I think there's an issue with intra-process subscription handling in the lifecycle node case, where we're destroying the bond connection in the on_deactivate callback. I'm not 100% sure the root cause (yet, will comment below if it becomes more clear to me), but what is clear to me from the backtrace is that it goes down into the IPC pipeline, as I see many intra- prefixed methods deeper into the trace. See stage 15, where it starts on dispatch_intra_process - in 22, we see rclcpp::Experimental::SubscriptionIntraProcess.
Steps to reproduce issue
We actually have a nice unit test that is pretty concise and relies almost exclusively on standard rclcpp / bond APIs in Nav2: https://github.com/ros-navigation/navigation2/blob/main/nav2_lifecycle_manager/test/test_bond.cpp#L128-L169. Pick any of the nav2_system_tests/src/system as well to reproduce. Or simply launch Nav2 and try to control+C out of it.
We have our on_deactivate method reset the bond connection https://github.com/ros-navigation/navigation2/blob/main/nav2_docking/opennav_docking/src/docking_server.cpp#L126-L144 who is implemented in Nav2's lifecycle node wrapper that is shared by all nodes https://github.com/ros-navigation/navigation2/blob/main/nav2_util/src/lifecycle_node.cpp#L146-L155
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.