GCP us-central1-b incident: entrypoint served global 429 "Rate exceeded." from 16:16 UTC 2026-09-01; no monitoring caught it

What happened

On 2026-09-01 ~16:16–16:31 UTC, the production Cloud Run entrypoint (policyengine-simulation-entry, project policyengine-simulation-entry) returned an instant 429 Rate exceeded. (Google Frontend, ~0.1s) for every request on every path (/health, /docs, /versions/*). App-v2 UK society-wide reports failed with a 500 wrapping the upstream 429 from GET /versions/policyengine (v1 API SimulationEntrypointClient.resolve_app_name).

Evidence gathered during the incident:

• https://simulation.api.policyengine.org/health → 429 Rate exceeded. (via the LB)
• https://policyengine-simulation-entry-4xy55an4yq-uc.a.run.app/health → 429 (direct stable URL, bypasses the LB/Cloud Armor) — so this was Cloud Run itself refusing admission, not an edge policy
• Tagged revision URL prod-10---… → 429 as well
• policyengine-simulation-entry-staging (same project/region) → 200 (5.5s cold start) — rules out project-wide quota/billing
• Modal gateway …-web-app.modal.run/health → 200 — backend healthy
• No deploy in the window (last: run 33270913487, 2026-08-29, same SHA as main)

An instant service-scoped 429 with that body is Cloud Run's admission-control refusal: all instances busy at max capacity with a full request queue (prod: --concurrency 80 --max-instances 20), i.e. either a large request burst or wedged/slow instances collapsing effective capacity. The service recovered on its own at ~16:31 UTC before any intervention reached prod (a re-deploy of the same SHA was dispatched as an instance refresh — run 33531984855 — and the recovery preceded its prod phase). A UK society-wide calculation (/uk/economy/82209/over/1?region=uk&time_period=2026) completed end-to-end right after.

Follow-ups this issue asks for

1. Forensics (needs GCP access): Cloud Run metrics/logs for policyengine-simulation-entry (us-central1) around 16:00–16:35 UTC — request rate, container instance count, request latencies, 429 count, and the request log by client identity/path just before onset. Was this a burst (from which caller?) or instances wedged behind a slow Modal upstream (25s proxy timeout × 80 concurrency can hold a lot of slots)?
2. Uptime monitoring: there is no Better Stack (or equivalent) monitor on https://simulation.api.policyengine.org/health — the v1 API host has one, this host does not; nobody was paged and the outage was found by a user (report page failure). Add a monitor.
3. Capacity review: whether --max-instances 20 × --concurrency 80 and the 25s upstream timeout compose safely when the Modal gateway is cold/slow; consider a lower per-request admission ceiling or faster shed for /versions/* (cacheable, tiny) so version resolution doesn't die with job traffic.

Related context: the July bot-wave incidents on the v1 API LB (docs/migration in policyengine-api, lb-cloud-armor-runbook.md) — but note Cloud Armor could not have caused this one (direct URL reproduced it), and the sim LB policy state should be checked separately if forensics point to a flood.

Start with Cloud Run metrics and request logs for policyengine-simulation-entry in us-central1 from 16:00–16:35 UTC, including request rate, instances, latency, 429s, and caller/path data. Check the /health endpoint and the existing v1 API monitor, then review the 25s upstream timeout with --concurrency 80 and --max-instances 20. Done means documenting the incident cause, adding uptime monitoring, and recording the capacity decision.