ros2 / ros2/launch

Parameterized `launch_pytest` tests don't close node log files

Open
#793 5 comments 0 reactions 1 assignee View on GitHub

@wjwwood is already working on this.

Since Aug 22, 2024.

Dominant language
Python
Stars
155
Forks
182
Avg merge
2d 14h
Merged PRs (30d)
6

Description

Bug report

When @pytest.mark.parametrize is used in a function scope with launch_pytest the node log files from each test parameterization remain open even when execution of that specific parameterized test is complete. For large projects with dozens or hundreds of nodes this eventually causes a OSError: [Errno 24] Too many open files error.

Required Info:

  • Operating System: Ubuntu Linux 20.04 amd64
  • Installation type: source
  • Version or commit hash: 9e517a3abd016367244fff9fa96ac8eae371a00b
  • DDS implementation: RTI Connext
  • Client library (if applicable): rclpy
Steps to reproduce issue

The code below cannot be run directly, but illustrates the issue. The TestContext object represents a specific test setup we want to run, and is constructed by name from a list of acceptable names in ALL_TEST_NAMES. This allows us to cherry-pick a subset of tests to run from our entire suite, with pytest -k or pytest --regex. The idea is each tests is run in sequence, against a new execution context and saved the results to a TEST_RESULTS data structure in the global context. When the test session ends, the AnalyzeResults function is called to analyze all results from all tests.

import pytest

import launch
import launch_pytest

from my_project.test_framework import TestContext, ALL_TEST_NAMES, AnalyzeResults

TEST_RESULTS = []

@pytest.fixture(scope='function')
def test_context(request):
    test_context = TestContext(launch_test=request.param)
    test_context.do_some_preprocessing_steps()
    return test_context

@launch_pytest.fixture(scope='function')
def launch_description(test_context):
    return launch.LaunchDescription(
        test_context.get_all_actions() + [launch_pytest.actions.ReadyToTest()])

@pytest.mark.parametrize('test_context', ALL_TEST_NAMES, indirect=True)
@pytest.mark.launch(fixture=launch_description)
def test_execution(test_context):
    results = test_context.do_some_post_postprocessing_steps()
    TEST_RESULTS.append(results)

@pytest.fixture(scope='session', autouse=True)
def run_assertions():
    yield
    AnalyzeResults(TEST_RESULTS)
Expected behavior

Test completes without throwing an error.

Actual behavior

Python throws an internal error OSError: [Errno 24] Too many open files.

This only happens several tests into the sequence, pointing to the fact that something internally is not cleaning up after itself. I have painfully tracked lsof -p <pid_of_pytest_invocation> over time and realized that the ROS2 node log files created by each parameterized test are not being closed, even though the stack has shut down. So, let's say we have k nodes and therefore k log files created by ROS per launch test that is run. If I run n parameterized tests sequentially the number of log files open at test i, where i is in {1 ... n}, will be i * k.

Additional information

You can always increase the number of files that may be opened at once with ulimit but this is a bandaid and not a solution.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.