PolicyEngine / PolicyEngine/cliff-watch

Intermittent 'rules-engine formulas must be deterministic' errors from /api/series

Open
#38 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Twice on 2026-07-03 (Europe/Brussels morning), in-app series calculations against the deployed API failed with policyengine-core's randomness guard, e.g.:

Calculation failed: The formula for 'slcsp_age_0' called numpy.random.seed(), but rules-engine formulas must be deterministic functions of their inputs. Remove the random call. If you need a stochastic input, compute it once when building the dataset (with a seeded generator) and store it as an input variable instead.

and the same wording naming age_head during a cliff-refinement request. Both appeared in the frontend as "Calculation failed: …" (the api/* error prefix), for an NC household (adult 30, children 4 and 8, ZIP 27601, all programs) swept 0–100k.

The guard landed in policyengine-core on 2026-06-04 (randomness_guard.py, "Forbid randomness inside variable formulas"); this repo floats policyengine-us>=1.715.2, so deployments resolve whatever us/core pair exists at build time.

What I could not do is reproduce it on demand: direct POSTs to /api/series on the current production deployment succeed for the same household — full 0–100k sweep at step 1000, refinement-shaped requests with min_earned_income, and ACA-region ranges where slcsp_age_0 computes (aca_ptc > 0). That leaves an intermittent trigger — plausibly a conditional randomness path (e.g., a take-up or imputation branch) that only some request shapes or instance states reach.

Suggestions:

  • Pin policyengine-us (and transitively policyengine-core) to an exact known-good pair in requirements.txt/pyproject.toml so deploys stop drifting, and bump deliberately.
  • Grep the pinned policyengine-us for np.random reachable from formulas (slcsp_age_0, age_head dependency chains) and either fix upstream or pin below the offending release.
  • If it recurs, capture the full traceback server-side — the guard's error names the innermost formula but not the caller that seeded.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the /api/series entry point and inspect the resolved policyengine-us/policyengine-core versions in requirements.txt or pyproject.toml. Read randomness_guard.py, then grep the pinned policyengine-us dependency chains for np.random near slcsp_age_0 and age_head; if the issue recurs, capture the server-side traceback. Done means the offending dependency pair or upstream formula is identified and the deployment no longer reaches nondeterministic formulas.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.