ros2 / ros2/rclcpp

Exiting a subscriber causes subsequent subscribers to no longer receive messages

Open
#1,831 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

more-information-needed
Dominant language
C++
Stars
805
Forks
564
Avg merge
1d 17h
Merged PRs (30d)
27

Description

Bug report

Exiting a subscriber (ex. "ros2 topic echo /tf") causes subsequent subscribers to no longer receive messages. Pre-existing subscribers continue to receive messages ok.

The behavior is truly strange. In the attached video I show:

  1. Running ros2 topic echo /tf subscriber
  2. Running a c++ node that publishes /tf data
    (data is then seen in [1] subscriber, no issue)
  3. Exiting subscriber in [1] (problem seems to arise here)
  4. Running subscriber in [1] again and no data is streamed.
  5. Showing Rviz is still running and receiving messages since it was a pre-existing subscriber. Also running two ros2 topic echo /tf subscribers and exiting only one of them, the other will continue to see data after issue presents.

I also show that:

  1. ros2 node info <tf_node_publisher> cannot find node after issue presents, but...
    • node is indeed running and clearly still publishing both /tf and on other topics. This points to a possibly discovery issue?
    • /tf will show 0 publishers, but pre-existing subscribers are still getting published data (huh?!?!)

Required Info:

  • Operating System:
    • Ubuntu 20.04
  • Installation type:
    • binaries from foxy
  • Version or commit hash:
  • DDS implementation:
    • Fast-RTPS
  • Client library (if applicable):
    • rclcpp
Steps to reproduce issue

I don't yet have enough information to provide a reliable way to reproduce this problem.

Expected behavior
  • Order of publisher/subscriber node startup should not matter
  • Existing a subscriber (such as ros2 cli echo) should not trash operational state of another node
Actual behavior
  • Subscribers started after publisher node receive no messages
  • Subscribers started before publisher node do continue to receive messages
Video demo

See youtube video of behavior here:
https://www.youtube.com/watch?v=6YKKcrbBR8A

  • Intro to issue ends @2:00
  • Plot thickens @4:00
  • Somewhat of a conclusion @9:00
Related Issues

May possibly related to these recent FastRTPS issues:

Additional information
  • I also encountered startup issues with another rclpy-based python node in Aug/Sept. This required restarting all nodes, waiting a minute for ros2 node list to clear out and relaunching the system. This started after ros2 repo updates. It may be related since it seemed to be a similar node discovery issue.
  • It's possible this issue is something in my c++ node, but given the circumstances and evidence so far I suspect a recent code change to rclcpp or FastRTPS library. Rclcpp/FastRTPS has been practically flawless since 2019 when I started using it until recently when I started seeing this weird behavior after some upgrades.
  • I am going to try Connext middleware and see if the problem persists

Any other suggestions to investigate further?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the sequence with ros2 topic echo /tf, the C++ publisher, and Fast-RTPS, then compare the behavior with Connext as suggested. Inspect the rclcpp and Fast-RTPS discovery path, using the linked FastRTPS issues as context. Done means a reliable reproduction and a confirmed cause or fix for subscribers losing messages after one exits.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.