PolicyEngine / PolicyEngine/policyengine-app-v2

Quarantine or fix 40 localStorage.clear test failures in app/ migration suite

Open
#998 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
HTML
Stars
1
Forks
3
Avg merge
1d 50m
Merged PRs (30d)
39

Description

Summary

Several test failures are known-broken on main across repos but are neither fixed nor quarantined. Every PR body ends up noting "these failures predate this PR" instead of the suite being green. This makes real regressions harder to spot.

What goes wrong

policyengine-api

  • tests/to_refactor/python/test_us_policy_macro.py::test_utah hangs locally; module references policyengine_api/worker.py that no longer exists. Confirmed on master by multiple agents.

policyengine-api-v2 (simulation project)

  • tests/test_household_calculation.py — macOS-only Blosc2 import failures.
  • tests/test_agent.pyAskRequest ImportError.

policyengine-app-v2

  • 40 failures in app/src/tests/unit/libs/migration/* from localStorage.clear is not a function.

Suggested fix

Per repo, either:

  1. Fix the underlying issue (preferred), or
  2. Quarantine with an explicit marker so the default suite is green:
# pytest
@pytest.mark.skip(reason="Broken on main — tracked in #NNNN, fix pending")
def test_utah(): ...

# vitest
it.skip('migration test — localStorage.clear not a function, tracked in #NNNN', ...)
  1. Surface via a separate CI job (pytest -m broken) that's allowed to fail, so the status is visible without masking real regressions.

The anti-pattern to eliminate: PR authors noting "these failures predate this PR" in every review. If they predate every PR, they should be marked so CI reflects that.

Severity

Medium (dev-experience). Hides real regressions, creates review friction, breaks the "CI green = merge" invariant.

Scope

Filing this in three repos: policyengine-api, policyengine-api-v2, policyengine-app-v2.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the named suites, especially app/src/tests/unit/libs/migration/*, tests/test_household_calculation.py, tests/test_agent.py, and tests/to_refactor/python/test_us_policy_macro.py::test_utah. Review each failure and decide whether it can be fixed or should be quarantined with the proposed test marker. Done means the default suites are green while broken tests remain visible through an allowed-to-fail or separately marked path.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, python
Domain
developer-experience, frontend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.