PolicyEngine / PolicyEngine/policyengine-us

CI can miss failures against newer policyengine-core because uv.lock pins stale core

Open
#8,753 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
162
Forks
212
Avg merge
3d 3h
Merged PRs (30d)
97

Description

## Summary

Household API staging deployment tests recently surfaced a `policyengine-core` compatibility failure that `policyengine-us` CI did not catch. The likely reason is that `policyengine-us` test jobs install from the committed `uv.lock`, which currently pins an older `policyengine-core`, while downstream consumers can resolve newer compatible `policyengine-core` versions from the package's lower-bound dependency range.

## Observed downstream failure

The household API staging tests failed with:

```text
The formula for 'is_ssi_recipient_for_medicaid' called numpy.random.seed(), but rules-engine formulas must be deterministic...
```

That failure appeared in the household API deployment path after resolving newer package versions, including a newer `policyengine-core` version than the one used by the locked `policyengine-us` test environment.

Related temporary downstream pin:

- PolicyEngine/policyengine-household-api#1575
- PolicyEngine/policyengine-household-api#1576

## Why this matters for policyengine-us

`policyengine-us` declares a compatible lower-bound dependency on `policyengine-core`, but normal CI runs appear to use `uv sync --extra dev`, which respects the committed lock file. In the current checkout, `uv.lock` pins `policyengine-core==3.26.0`, so PR/push tests can pass against that stale core even when downstream installs resolve a newer compatible `policyengine-core` that changes runtime behavior.

This means `policyengine-us` CI may not catch regressions introduced by newer `policyengine-core` versions until a downstream application, such as `policyengine-household-api`, installs and exercises that newer core.

## Suggested fixes

- Add a CI job that tests against the latest compatible `policyengine-core`, independent of the committed lock file.
- Consider a scheduled lock refresh/upgrade workflow that validates the package against upgraded dependencies.
- Add a focused regression test for the Medicaid/SSI path that triggered the household API failure, especially around `is_ssi_recipient_for_medicaid`.
- Consider porting or mirroring relevant household API partner/customer-input fixtures into `policyengine-us` when they exercise package behavior that normal YAML tests miss.
- If newer `policyengine-core` behavior is intentionally incompatible, add an upper bound or compatibility gate until `policyengine-us` is updated.

## Notes

The failing formula itself does not obviously contain a direct random seed call in the current source. This issue is about the test-environment gap first: `policyengine-us` should catch compatibility failures against newer allowed `policyengine-core` versions before they reach downstream deployments.

Contributor guide

Open the contributing guide

Research direction

Start with uv.lock and the CI job that runs `uv sync --extra dev`; compare its pinned policyengine-core with the compatible versions allowed by policyengine-us. Review the reported `is_ssi_recipient_for_medicaid` failure and existing test coverage, then define completion as CI exercising a newer compatible core and catching the relevant compatibility regression before downstream deployment.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.