PolicyEngine / PolicyEngine/policyengine-us

Prevent selective test failures for changed fixture modules

Open
#9,139 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
162
Forks
212
Avg merge
3d 3h
Merged PRs (30d)
97

Description

## Problem

The Quick Feedback selective-test runner treats every Python, YAML, or YML file under `policyengine_us/tests/` as an executable test target.

When a changed file is a fixture or helper module with no tests, the runner invokes it directly. Pytest collects zero tests and exits with code 5. The selective runner preserves that code and fails the job even when every actual selected test passes.

This occurred on #9138 after an annotation-only change to:

`policyengine_us/tests/microsimulation/data/fixtures/test_extend_single_year_dataset.py`

The affected job is: https://github.com/PolicyEngine/policyengine-us/actions/runs/29957280966/job/89049412054

In that run, the actual selected suites passed 173 tests, but the fixture module collected zero tests and caused the final failure.

## Root cause

`SelectiveTestRunner.is_test_file` classifies files based only on being beneath `policyengine_us/tests/` and having a supported source suffix. `map_files_to_tests` then adds those files directly, and `run_tests` propagates any nonzero subprocess result.

## Proposed direction

- Distinguish executable tests from fixture and helper modules.
- Map changed fixtures/helpers to their consuming tests or another appropriate test target.
- Do not broadly ignore pytest exit code 5, because that could hide genuine test-selection mistakes.

## Acceptance criteria

- A changed fixture/helper module does not cause Quick Feedback to fail solely because it collects no tests.
- Relevant consuming tests are still selected where practical.
- Genuine test collection and execution failures remain fatal.
- Add regression coverage to `policyengine_us/tests/core/test_run_selective_tests.py` for fixture-only changes.

Contributor guide

Open the contributing guide

Research direction

Start with SelectiveTestRunner.is_test_file, map_files_to_tests, and run_tests, then read policyengine_us/tests/core/test_run_selective_tests.py. Reproduce the fixture-only change case and add regression coverage there. Done means fixture or helper changes do not fail solely on zero collected tests, consuming tests are selected where practical, and genuine collection or execution failures remain fatal.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.