ros2 / ros2/system_tests

Flaky test test_n_nodes__rmw_fastrtps_cpp

Open
#486 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

backlog
Dominant language
C++
Stars
45
Forks
57
Avg merge
9h 19m
Merged PRs (30d)
2

Description

Bug report

Required Info:

  • Operating System:
    • Ubuntu Focal
  • Installation type:
    • source
  • Version or commit hash:
    • fab8e95bcfa4ac21415c610068cc14ea0ff7ad78
  • DDS implementation:
    • FastDDS
  • Client library (if applicable):
    • rclcpp
Actual behavior
projectroot.test_n_nodes__rmw_fastrtps_cpp
test_rclcpp.TestNNodesAfterShutdown.test_10_nodes

The test appears to complete successfully, but then fails to shutdown afterwards. launch escalates to a SIGTERM signal.

https://ci.ros2.org/job/nightly_linux_repeated/2416

Console output 
      Start 70: test_n_nodes__rmw_fastrtps_cpp

70: Test command: /home/jenkins-agent/workspace/nightly_linux_repeated/venv/bin/python3.8 "-u" "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/ament_cmake_test/share/ament_cmake_test/cmake/run_test.py" "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp/test_results/test_rclcpp/test_n_nodes__rmw_fastrtps_cpp.xunit.xml" "--package-name" "test_rclcpp" "--output-file" "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp/launch_test/test_n_nodes__rmw_fastrtps_cpp.txt" "--env" "RCL_ASSERT_RMW_ID_MATCHES=rmw_fastrtps_cpp" "RMW_IMPLEMENTATION=rmw_fastrtps_cpp" "--append-env" "LD_LIBRARY_PATH=/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp" "--command" "/home/jenkins-agent/workspace/nightly_linux_repeated/venv/bin/python3" "-m" "launch_testing.launch_test" "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp/test_n_nodes__rmw_fastrtps_cpp_Release.py" "--junit-xml=/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp/test_results/test_rclcpp/test_n_nodes__rmw_fastrtps_cpp.xunit.xml" "--package-name=test_rclcpp"
70: Test timeout computed to be: 60
70: -- run_test.py: extra environment variables:
70:  - RCL_ASSERT_RMW_ID_MATCHES=rmw_fastrtps_cpp
70:  - RMW_IMPLEMENTATION=rmw_fastrtps_cpp
70: -- run_test.py: extra environment variables to append:
70:  - LD_LIBRARY_PATH+=/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp
70: -- run_test.py: invoking following command in '/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp':
70:  - /home/jenkins-agent/workspace/nightly_linux_repeated/venv/bin/python3 -m launch_testing.launch_test /home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp/test_n_nodes__rmw_fastrtps_cpp_Release.py --junit-xml=/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp/test_results/test_rclcpp/test_n_nodes__rmw_fastrtps_cpp.xunit.xml --package-name=test_rclcpp
70: [INFO] [launch]: All log files can be found below /home/rosbuild/.ros/log/2021-09-20-14-45-01-952454-3241f705c46b-406396
70: [INFO] [launch]: Default logging verbosity is set to INFO
70: test_10_nodes (test_rclcpp.TestNNodes) ... [INFO] [node_with_name_0-1]: process started with pid [406399]
70: [INFO] [node_with_name_1-2]: process started with pid [406401]
70: [INFO] [node_with_name_2-3]: process started with pid [406403]
70: [INFO] [node_with_name_3-4]: process started with pid [406405]
70: [INFO] [node_with_name_4-5]: process started with pid [406407]
70: [INFO] [node_with_name_5-6]: process started with pid [406441]
70: [INFO] [node_with_name_6-7]: process started with pid [406445]
70: [INFO] [node_with_name_7-8]: process started with pid [406447]
70: [INFO] [node_with_name_8-9]: process started with pid [406457]
70: [INFO] [node_with_name_9-10]: process started with pid [406470]
70: [INFO] [node_check_names-11]: process started with pid [406479]
70: [INFO] [node_check_names-11]: process has finished cleanly [pid 406479]
70: [INFO] [node_with_name_9-10]: sending signal 'SIGINT' to process[node_with_name_9-10]
70: [INFO] [node_with_name_8-9]: sending signal 'SIGINT' to process[node_with_name_8-9]
70: [INFO] [node_with_name_7-8]: sending signal 'SIGINT' to process[node_with_name_7-8]
70: [INFO] [node_with_name_6-7]: sending signal 'SIGINT' to process[node_with_name_6-7]
70: [INFO] [node_with_name_5-6]: sending signal 'SIGINT' to process[node_with_name_5-6]
70: [INFO] [node_with_name_4-5]: sending signal 'SIGINT' to process[node_with_name_4-5]
70: [INFO] [node_with_name_3-4]: sending signal 'SIGINT' to process[node_with_name_3-4]
70: [INFO] [node_with_name_2-3]: sending signal 'SIGINT' to process[node_with_name_2-3]
70: [INFO] [node_with_name_1-2]: sending signal 'SIGINT' to process[node_with_name_1-2]
70: [INFO] [node_with_name_0-1]: sending signal 'SIGINT' to process[node_with_name_0-1]
70: [node_with_name_9-10] [INFO] [1632149102.241335443] [rclcpp]: signal_handler(signal_value=2)
70: [node_with_name_8-9] [INFO] [1632149102.245046493] [rclcpp]: signal_handler(signal_value=2)
70: [node_with_name_7-8] [INFO] [1632149102.247834727] [rclcpp]: signal_handler(signal_value=2)
70: [node_with_name_6-7] [INFO] [1632149102.250542139] [rclcpp]: signal_handler(signal_value=2)
70: [node_with_name_5-6] [INFO] [1632149102.252687954] [rclcpp]: signal_handler(signal_value=2)
70: [node_with_name_4-5] [INFO] [1632149102.255035717] [rclcpp]: signal_handler(signal_value=2)
70: [node_with_name_3-4] [INFO] [1632149102.264126567] [rclcpp]: signal_handler(signal_value=2)
70: [node_with_name_2-3] [INFO] [1632149102.266437111] [rclcpp]: signal_handler(signal_value=2)
70: [node_with_name_1-2] [INFO] [1632149102.269126543] [rclcpp]: signal_handler(signal_value=2)
70: [node_with_name_0-1] [INFO] [1632149102.270640193] [rclcpp]: signal_handler(signal_value=2)
70: [INFO] [node_with_name_7-8]: process has finished cleanly [pid 406447]
70: [INFO] [node_with_name_5-6]: process has finished cleanly [pid 406441]
70: [INFO] [node_with_name_8-9]: process has finished cleanly [pid 406457]
70: [INFO] [node_with_name_9-10]: process has finished cleanly [pid 406470]
70: [INFO] [node_with_name_6-7]: process has finished cleanly [pid 406445]
70: [INFO] [node_with_name_4-5]: process has finished cleanly [pid 406407]
70: [INFO] [node_with_name_3-4]: process has finished cleanly [pid 406405]
70: [INFO] [node_with_name_2-3]: process has finished cleanly [pid 406403]
70: [INFO] [node_with_name_0-1]: process has finished cleanly [pid 406399]
70: [ERROR] [node_with_name_1-2]: process[node_with_name_1-2] failed to terminate '5' seconds after receiving 'SIGINT', escalating to 'SIGTERM'
70: [INFO] [node_with_name_1-2]: sending signal 'SIGTERM' to process[node_with_name_1-2]
70: [ERROR] [node_with_name_1-2]: process has died [pid 406401, exit code -15, cmd '/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp/node_with_name'].
70: ok
70: 
70: ----------------------------------------------------------------------
70: Ran 1 test in 0.270s
70: 
70: OK
70: test_10_nodes (test_rclcpp.TestNNodesAfterShutdown) ... FAIL
70: 
70: ======================================================================
70: FAIL: test_10_nodes (test_rclcpp.TestNNodesAfterShutdown)
70: ----------------------------------------------------------------------
70: Traceback (most recent call last):
70:   File "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp/test_n_nodes__rmw_fastrtps_cpp_Release.py", line 61, in test_10_nodes
70:     launch_testing.asserts.assertExitCodes(
70:   File "/home/jenkins-agent/workspace/nightly_linux_repeated/ws/install/launch_testing/lib/python3.8/site-packages/launch_testing/asserts/assert_exit_codes.py", line 62, in assertExitCodes
70:     assert info.returncode in allowable_exit_codes, 'Proc {} exited with code {}'.format(
70: AssertionError: Proc node_with_name_1-2 exited with code -15
70: 
70: ----------------------------------------------------------------------
70: Ran 1 test in 0.001s
70: 
70: FAILED (failures=1)
70: -- run_test.py: return code 1
70: -- run_test.py: verify result file '/home/jenkins-agent/workspace/nightly_linux_repeated/ws/build/test_rclcpp/test_results/test_rclcpp/test_n_nodes__rmw_fastrtps_cpp.xunit.xml'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with test_rclcpp.TestNNodesAfterShutdown and the test_n_nodes__rmw_fastrtps_cpp failure in the linked CI run. Reproduce the shutdown behavior with rmw_fastrtps_cpp and inspect why one node requires SIGTERM; done means the test consistently exits with an allowed code without escalation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.