PolicyEngine / PolicyEngine/policyengine-household-api
Add PR-level smoke checks for lightweight gateway runtimes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 3
- Avg merge
- 6h 34m
- Merged PRs (30d)
- 8
Description
Problem
A code-only deploy can pass PR CI but fail later in Release to Modal because PR CI runs tests in the full development environment, while release deploys lightweight gateway runtimes with smaller dependency sets.
This happened in the 0.25.0 observability release. PR CI passed, but the release run failed in staging:
- Release run: https://github.com/PolicyEngine/policyengine-household-api/actions/runs/28386574895
Integration tests against Modal stagingtimed out loading/versions/us.Deploy Cloud Run failover to stagingdeployed the gateway but/liveness_checkreturned HTTP 503.
The deployed logs showed startup import failures:
- Modal gateway:
ModuleNotFoundError: No module named 'numpy'- Import path:
modal_release/gateway_app.py->modal_release/gateway.py->observability/flask.py->utils.config_loader->utils/__init__.py->utils/json.py. utils/json.pyimports NumPy, but the Modal gateway image is intentionally lightweight and does not install NumPy.
- Import path:
- Cloud Run gateway:
ModuleNotFoundError: No module named 'policyengine_observability'failover/cloud_run_gateway.pyimports observability code, butgcp/cloud_run/gateway.Dockerfiledid not installpolicyengine-observability[flask].
Why PR CI missed it
.github/workflows/pr.yml installs the full project and runs unit tests. That environment includes dependencies that the lightweight gateway images do not include, so import errors in the deployable artifacts were masked.
The release workflow is the first place that actually builds/deploys the minimal Modal gateway and Cloud Run gateway runtimes and checks their endpoints.
There is also a smaller drift: PR CI uses Python 3.12, while release uses Python 3.13.
Proposed safeguard
Add targeted PR-level smoke checks for deployable runtime bootability. These should be path-triggered and cheap, not a full staging release.
Suggested checks:
-
Modal gateway minimal import smoke
- Create an isolated environment matching
household_api_gateway_image()dependencies. - Import
policyengine_household_api.modal_release.gateway_apporpolicyengine_household_api.modal_release.gateway. - This would catch imports that pull in dependencies missing from the lightweight Modal gateway image.
- Create an isolated environment matching
-
Cloud Run gateway Docker boot smoke
- Build
gcp/cloud_run/gateway.Dockerfile. - Run the image locally in CI.
- Curl
/liveness_check. - This would catch missing Dockerfile dependencies before merge.
- Build
-
Align PR CI Python with release Python where possible (
3.13).
Suggested path triggers include:
policyengine_household_api/modal_release/**
policyengine_household_api/failover/**
policyengine_household_api/observability/**
policyengine_household_api/utils/**
gcp/cloud_run/gateway.Dockerfile
gcp/cloud_run/gateway_start.sh
.github/scripts/cloud-run-deploy-failover.sh
.github/workflows/deploy-staged.yml
Non-goals
This should not run the full Release to Modal workflow on every PR. The goal is only to catch basic artifact boot/import failures before merge. Release CI should still own real staging deploys, deployed integration tests, Cloud Run fallback validation, and production gating.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with .github/workflows/pr.yml, household_api_gateway_image(), modal_release/gateway_app.py, gcp/cloud_run/gateway.Dockerfile, and the listed path triggers to understand existing CI and runtime dependencies. Add cheap PR checks that import the minimal Modal gateway and build/run the Cloud Run image, confirming /liveness_check; align PR Python with release Python where possible. Done means these checks run only for relevant changes and catch missing runtime dependencies without running the full release workflow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, python
- Domain
- ci-cd, cloud, devops, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100