PolicyEngine / PolicyEngine/policyengine-household-api

Add PR-level smoke checks for lightweight gateway runtimes

Open
#1,583 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1
Forks
3
Avg merge
6h 34m
Merged PRs (30d)
8

Description

Problem

A code-only deploy can pass PR CI but fail later in Release to Modal because PR CI runs tests in the full development environment, while release deploys lightweight gateway runtimes with smaller dependency sets.

This happened in the 0.25.0 observability release. PR CI passed, but the release run failed in staging:

The deployed logs showed startup import failures:

  • Modal gateway: ModuleNotFoundError: No module named 'numpy'
    • Import path: modal_release/gateway_app.py -> modal_release/gateway.py -> observability/flask.py -> utils.config_loader -> utils/__init__.py -> utils/json.py.
    • utils/json.py imports NumPy, but the Modal gateway image is intentionally lightweight and does not install NumPy.
  • Cloud Run gateway: ModuleNotFoundError: No module named 'policyengine_observability'
    • failover/cloud_run_gateway.py imports observability code, but gcp/cloud_run/gateway.Dockerfile did not install policyengine-observability[flask].

Why PR CI missed it

.github/workflows/pr.yml installs the full project and runs unit tests. That environment includes dependencies that the lightweight gateway images do not include, so import errors in the deployable artifacts were masked.

The release workflow is the first place that actually builds/deploys the minimal Modal gateway and Cloud Run gateway runtimes and checks their endpoints.

There is also a smaller drift: PR CI uses Python 3.12, while release uses Python 3.13.

Proposed safeguard

Add targeted PR-level smoke checks for deployable runtime bootability. These should be path-triggered and cheap, not a full staging release.

Suggested checks:

  1. Modal gateway minimal import smoke

    • Create an isolated environment matching household_api_gateway_image() dependencies.
    • Import policyengine_household_api.modal_release.gateway_app or policyengine_household_api.modal_release.gateway.
    • This would catch imports that pull in dependencies missing from the lightweight Modal gateway image.
  2. Cloud Run gateway Docker boot smoke

    • Build gcp/cloud_run/gateway.Dockerfile.
    • Run the image locally in CI.
    • Curl /liveness_check.
    • This would catch missing Dockerfile dependencies before merge.
  3. Align PR CI Python with release Python where possible (3.13).

Suggested path triggers include:

policyengine_household_api/modal_release/**
policyengine_household_api/failover/**
policyengine_household_api/observability/**
policyengine_household_api/utils/**
gcp/cloud_run/gateway.Dockerfile
gcp/cloud_run/gateway_start.sh
.github/scripts/cloud-run-deploy-failover.sh
.github/workflows/deploy-staged.yml

Non-goals

This should not run the full Release to Modal workflow on every PR. The goal is only to catch basic artifact boot/import failures before merge. Release CI should still own real staging deploys, deployed integration tests, Cloud Run fallback validation, and production gating.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .github/workflows/pr.yml, household_api_gateway_image(), modal_release/gateway_app.py, gcp/cloud_run/gateway.Dockerfile, and the listed path triggers to understand existing CI and runtime dependencies. Add cheap PR checks that import the minimal Modal gateway and build/run the Cloud Run image, confirming /liveness_check; align PR Python with release Python where possible. Done means these checks run only for relevant changes and catch missing runtime dependencies without running the full release workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, python
Domain
ci-cd, cloud, devops, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.