PolicyEngine / PolicyEngine/policyengine-us
Prevent selective test failures for changed fixture modules
- Dominant language
- Python
- Stars
- 162
- Forks
- 212
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 97
Description
## Problem
The Quick Feedback selective-test runner treats every Python, YAML, or YML file under `policyengine_us/tests/` as an executable test target.
When a changed file is a fixture or helper module with no tests, the runner invokes it directly. Pytest collects zero tests and exits with code 5. The selective runner preserves that code and fails the job even when every actual selected test passes.
This occurred on #9138 after an annotation-only change to:
`policyengine_us/tests/microsimulation/data/fixtures/test_extend_single_year_dataset.py`
The affected job is: https://github.com/PolicyEngine/policyengine-us/actions/runs/29957280966/job/89049412054
In that run, the actual selected suites passed 173 tests, but the fixture module collected zero tests and caused the final failure.
## Root cause
`SelectiveTestRunner.is_test_file` classifies files based only on being beneath `policyengine_us/tests/` and having a supported source suffix. `map_files_to_tests` then adds those files directly, and `run_tests` propagates any nonzero subprocess result.
## Proposed direction
- Distinguish executable tests from fixture and helper modules.
- Map changed fixtures/helpers to their consuming tests or another appropriate test target.
- Do not broadly ignore pytest exit code 5, because that could hide genuine test-selection mistakes.
## Acceptance criteria
- A changed fixture/helper module does not cause Quick Feedback to fail solely because it collects no tests.
- Relevant consuming tests are still selected where practical.
- Genuine test collection and execution failures remain fatal.
- Add regression coverage to `policyengine_us/tests/core/test_run_selective_tests.py` for fixture-only changes.
Contributor guide
Research direction
Start with SelectiveTestRunner.is_test_file, map_files_to_tests, and run_tests, then read policyengine_us/tests/core/test_run_selective_tests.py. Reproduce the fixture-only change case and add regression coverage there. Done means fixture or helper changes do not fail solely on zero collected tests, consuming tests are selected where practical, and genuine collection or execution failures remain fatal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100