US production: /us/metadata returns 500 and state-region simulations fail with connection reset at the simulation gateway

Summary

The US v1 production stack is partially down, and monitoring can't see it because Better Stack only probes the API root (which returns 200):

1. GET /us/metadata returns HTTP 500 — observed on 2026-08-06, 2026-08-15, and 2026-08-21 (every check). /uk/metadata and / return 200. The app's US parameter pages depend on this endpoint.
2. US state-region society-wide simulations fail — GET /us/economy/{policy}/over/2?region=state/or&time_period=2027 returns {"status": "error", "message": "[Errno 104] Connection reset by peer", "result": null}, consistently on retry. Previously cached state results now error too (the cached result is gone and the recompute fails).
3. National runs hang — region=us requests sit in "computing" for 10+ minutes (normal completion is a few minutes).

As of 2026-08-21 18:00 UTC, every executor app in the Modal main environment (policyengine-simulation-py4-18-*, py4-20-3, py4-22-0) and policyengine-simulation-gateway show deployed with 0 running tasks while the API reports runs computing — requests are not starting on Modal. The reset-by-peer surfaces at the API → gateway → executor hop for the route production uses (resolved_app_name: policyengine-simulation-py4-20-3 on completed runs from before the failure).

Timeline (all observations read-only, from outside)

• 2026-07-17: state runs healthy (e.g. policy 98001 state/or 2027 → −$25.2M, stamped pe-us 1.764.6 / populace-us-2024-buildi-sparse… / policyengine-simulation-py4-20-3).
• 2026-08-06: /us/metadata 500; state runs still complete with the identical stamp.
• 2026-08-15: /us/metadata 500; state runs still complete.
• 2026-08-21: /us/metadata 500; state runs fail with connection reset; national hangs.

Better Stack logged only two brief auto-resolved root incidents (8/7 09:14–09:19, 8/14 04:47–04:52) in the same window.

Repro

curl -s -o /dev/null -w "%{http_code}\n" https://api.policyengine.org/us/metadata      # 500
curl -s -o /dev/null -w "%{http_code}\n" https://api.policyengine.org/uk/metadata      # 200
curl -s "https://api.policyengine.org/us/economy/98001/over/2?region=state/or&time_period=2027"
# {"status": "error", "message": "[Errno 104] Connection reset by peer", "result": null}

Possibly related

• The metadata failure window starts after the v1 persistence migration to SQLAlchemy/Alembic (#3788, 8/12) and Cloud Run revision changes in sim-api (#657, 7/28) — correlation only; I have not looked at logs.
• #2430 (simulation-API 500s surface as status: ok + result: null) describes the adjacent failure mode; this one at least surfaces as status: error.
• Suggest adding /us/metadata and a state-region simulation smoke to Better Stack — the root probe can't catch either.

User-facing impact: every state-level analysis on policyengine.org (US) currently fails; external partners sharing state reform links hit errors.

Start with production logs for GET /us/metadata and the API → gateway → Modal executor path, especially policyengine-simulation-py4-20-3, and compare them with the healthy UK endpoint. Trace why requests remain computing or return connection resets after the persistence migration and Cloud Run changes. Done means US metadata returns 200, state-region runs complete, national runs finish normally, and monitoring probes these paths.