Exiting a subscriber causes subsequent subscribers to no longer receive messages
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 805
- Forks
- 564
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 27
Description
Bug report
Exiting a subscriber (ex. "ros2 topic echo /tf") causes subsequent subscribers to no longer receive messages. Pre-existing subscribers continue to receive messages ok.
The behavior is truly strange. In the attached video I show:
- Running
ros2 topic echo /tfsubscriber - Running a c++ node that publishes /tf data
(data is then seen in [1] subscriber, no issue) - Exiting subscriber in [1] (problem seems to arise here)
- Running subscriber in [1] again and no data is streamed.
- Showing Rviz is still running and receiving messages since it was a pre-existing subscriber. Also running two
ros2 topic echo /tfsubscribers and exiting only one of them, the other will continue to see data after issue presents.
I also show that:
ros2 node info <tf_node_publisher>cannot find node after issue presents, but...- node is indeed running and clearly still publishing both /tf and on other topics. This points to a possibly discovery issue?
/tfwill show 0 publishers, but pre-existing subscribers are still getting published data (huh?!?!)
Required Info:
- Operating System:
- Ubuntu 20.04
- Installation type:
- binaries from foxy
- Version or commit hash:
- latest repo update
- deb [arch=amd64] http://packages.ros.org/ros2/ubuntu focal main
- DDS implementation:
- Fast-RTPS
- Client library (if applicable):
- rclcpp
Steps to reproduce issue
I don't yet have enough information to provide a reliable way to reproduce this problem.
Expected behavior
- Order of publisher/subscriber node startup should not matter
- Existing a subscriber (such as ros2 cli echo) should not trash operational state of another node
Actual behavior
- Subscribers started after publisher node receive no messages
- Subscribers started before publisher node do continue to receive messages
Video demo
See youtube video of behavior here:
https://www.youtube.com/watch?v=6YKKcrbBR8A
- Intro to issue ends @2:00
- Plot thickens @4:00
- Somewhat of a conclusion @9:00
Related Issues
May possibly related to these recent FastRTPS issues:
- FastRTPS 2299 Nodes don't reconnect to TCP discovery server
- FastRTPS 2318 TCP retransmission after deleted participant
Additional information
- I also encountered startup issues with another rclpy-based python node in Aug/Sept. This required restarting all nodes, waiting a minute for
ros2 node listto clear out and relaunching the system. This started after ros2 repo updates. It may be related since it seemed to be a similar node discovery issue. - It's possible this issue is something in my c++ node, but given the circumstances and evidence so far I suspect a recent code change to rclcpp or FastRTPS library. Rclcpp/FastRTPS has been practically flawless since 2019 when I started using it until recently when I started seeing this weird behavior after some upgrades.
- I am going to try Connext middleware and see if the problem persists
Any other suggestions to investigate further?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the sequence with ros2 topic echo /tf, the C++ publisher, and Fast-RTPS, then compare the behavior with Connext as suggested. Inspect the rclcpp and Fast-RTPS discovery path, using the linked FastRTPS issues as context. Done means a reliable reproduction and a confirmed cause or fix for subscribers losing messages after one exits.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100