PolicyEngine / PolicyEngine/policyengine-household-api

Cloud Run failover builds the current channel from the frontier lock

Open
#1,667 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1
Forks
3
Avg merge
6h 34m
Merged PRs (30d)
8

Description

What happened

The "Release to Modal" run for the 2026-09-16 weekly update (run 35130671170, main at 65b00500) failed at "Deploy Cloud Run failover to staging". Nothing reached production.

The failover worker image is built per channel from gcp/cloud_run/worker.Dockerfile: uv sync --frozen against the workspace lock, then uv pip install of that channel's country package on top. Today's lock resolves spm-calculator==1.0.0.post1 because frontier is policyengine-us==2.5.0. The current channel image then installs policyengine-us==1.824.6, which only declares spm-calculator>=0.2.0, so uv keeps 1.0.0.post1. But 1.824.6 imports spm_calculator.geoadj, removed in 1.0.0, and the preload step fails:

ModuleNotFoundError: No module named 'spm_calculator.geoadj'

Modal does not hit this because it never rebuilds current; it promotes last week's frontier app, whose image still has spm-calculator 0.3.1. Cloud Run is the only place that rebuilds both channels from one lock. This is the first rotation where a transitive dependency diverged between channels; the same build passed earlier the same day with 1.821.10 and 1.824.6.

Proposed fix

Treat each channel as a frozen artifact, the way Modal already does:

  • At release time, store the frontier's exported locked requirements (the uv export --frozen the Modal deploy already produces) with the release manifest, per channel. When a frontier is promoted to current, its requirements move with it.
  • The failover worker image installs from the channel's stored requirements instead of the workspace lock. Frontier gets today's export; current gets the one it was released with.
  • First-party household API packages remain the current checkout on both channels, as on Modal.
  • Bootstrap: seed the current current channel (1.824.6) from the lock at the pre-bump commit, since it was released before freezes existed.

This removes the need for anyone to think about companion pins on future policyengine-us, policyengine-core, or spm-calculator bumps.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with gcp/cloud_run/worker.Dockerfile and the Modal deploy flow that already runs uv export --frozen. Trace how release manifests and channel images are built, then define where each channel's exported requirements are stored and promoted. Done means frontier and current install their own frozen requirements, first-party household API packages still come from the checkout, and current is bootstrapped from the pre-bump lock.

Written by the indexing model from the issue text.

Assessment

Tech stack
dockerfile, gcp, python
Domain
build-system, cloud, devops, release
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.