PolicyEngine / PolicyEngine/policyengine-api
Quarantine or fix test_utah hang and other known-broken tests on master
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 18
- Forks
- 33
- Avg merge
- 23h 40m
- Merged PRs (30d)
- 22
Description
Summary
Several test failures are known-broken on main across repos but are neither fixed nor quarantined. Every PR body ends up noting "these failures predate this PR" instead of the suite being green. This makes real regressions harder to spot.
What goes wrong
policyengine-api
tests/to_refactor/python/test_us_policy_macro.py::test_utahhangs locally; module referencespolicyengine_api/worker.pythat no longer exists. Confirmed on master by multiple agents.
policyengine-api-v2 (simulation project)
tests/test_household_calculation.py— macOS-only Blosc2 import failures.tests/test_agent.py—AskRequestImportError.
policyengine-app-v2
- 40 failures in
app/src/tests/unit/libs/migration/*fromlocalStorage.clear is not a function.
Suggested fix
Per repo, either:
- Fix the underlying issue (preferred), or
- Quarantine with an explicit marker so the default suite is green:
# pytest
@pytest.mark.skip(reason="Broken on main — tracked in #NNNN, fix pending")
def test_utah(): ...
# vitest
it.skip('migration test — localStorage.clear not a function, tracked in #NNNN', ...)
- Surface via a separate CI job (
pytest -m broken) that's allowed to fail, so the status is visible without masking real regressions.
The anti-pattern to eliminate: PR authors noting "these failures predate this PR" in every review. If they predate every PR, they should be marked so CI reflects that.
Severity
Medium (dev-experience). Hides real regressions, creates review friction, breaks the "CI green = merge" invariant.
Scope
Filing this in three repos: policyengine-api, policyengine-api-v2, policyengine-app-v2.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/to_refactor/python/test_us_policy_macro.py::test_utah in policyengine-api, then inspect the named test files in policyengine-api-v2 and app/src/tests/unit/libs/migration in policyengine-app-v2. Run the affected pytest and Vitest tests to confirm the failures. Done means the underlying failures are fixed or explicitly quarantined, while the default suites are green and broken tests remain visible separately.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, python
- Domain
- developer-experience, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100