Gateway leaks Modal dispatch threads on hung control-plane calls; circuit breaker never opened during 2026-07-21 outage

Summary

On 2026-07-21 between ~01:02 and ~01:17 UTC, the production gateway returned 503 backend_unavailable for most /us/calculate requests on both channels. The Modal workers were healthy throughout (their logs show every request that reached them served with HTTP 200 in 1.5–4s, warm containers, min_containers=3 active). The outage was produced in the gateway's Modal dispatch path, and automatic failover to the Cloud Run fallback never triggered. Service was restored by a manual HOUSEHOLD_FAILOVER_FORCE_BACKEND=cloud_run deploy at 01:17:34, which replaced the gateway process; Modal-primary routing was restored later the same night and has been healthy since (full 6-household parity passed on both channels).

Timeline (UTC, from gateway Cloud Run logs, instance …1765a782)

Root cause

External trigger: starting ~01:02, a subset of the gateway's Modal SDK dispatches to Modal's control plane hung indefinitely — silent black-hole, no error. Success/failure was per-dispatch: the same gateway instance served 3–4s 200s and 90s 503s in the same minute, on both channels. (Consistent with silently dead gRPC connections in one of the two gunicorn worker processes; the 5-minute monitor's keep-alive connection stayed pinned to a healthy path and never failed. Worker-side logs confirm hung dispatches never reached the workers.)

Gateway defect 1 — no timeout on the Modal call itself. _call_modal_worker_dispatch (cloud_run_gateway.py) never passes timeout_seconds to call_modal_worker_dispatch, whose docstring warns that without it Modal waits indefinitely. The 90s bound exists only in _run_modal_operation's future.result(timeout=...).

Gateway defect 2 — permanent executor-slot leak. After the 90s timeout, future.cancel() is a no-op on a running thread, and the semaphore permit is released only by the done-callback, which never fires for a permanently hung call. Each hang permanently consumed one of the 32 MODAL_EXECUTOR_MAX_WORKERS slots. The arithmetic matches exactly: 8 + 24 = 32 hung dispatches, then modal_executor_saturated at 01:15:04. The catch-all route accelerated this by dispatching junk paths (/, /healthz, scanner probes) to Modal, burning slots.

Gateway defect 3 — circuit breaker blind to the failure mode. Zero modal_circuit_open events were logged. Health probes and the canary run on the dedicated _MODAL_PROBE_EXECUTOR with their own connections, which stayed healthy — so recovery checks kept passing while the request path bled out, and the circuit never opened. A partial-connection black-hole structurally defeats the canary-corroboration requirement; the outage lasted ~15 minutes instead of the ~1 minute the sliding-window thresholds intend.

Proposed fixes (in priority order)

1. Pass timeout_seconds (~90s) into call_modal_worker_dispatch from the gateway request path. The spawn+get(timeout) plumbing already exists in worker_dispatch.py. This converts a permanent slot leak into a 90s-bounded one and would have prevented the saturation phase entirely.
2. Let request-path failure evidence open the circuit when probes disagree. Sustained modal_request_failed volume over the sliding window should be able to open the channel circuit (or at least force fallback attempts) even when the canary passes, since probe connections can be healthy while request-path connections are not.
3. Stop dispatching unknown paths to Modal. Return 404 at the gateway for paths outside the known API surface instead of forwarding scanner noise through the dispatch executor.
4. (Hardening) Recycle gunicorn workers periodically (max_requests + jitter) or add Modal client connection health-checking, so silently dead control-plane connections self-heal without a deploy.

Evidence pointers

• Gateway logs: gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="household-api-production-gateway" AND timestamp>="2026-07-21T00:40:00Z" AND timestamp<="2026-07-21T01:25:00Z"' --project policyengine-household-api
• Worker logs (both channels healthy throughout): modal app logs for policyengine-household-api-uk2-88-18-us1-768-1 / -us1-771-2, window 00:45–01:20 UTC
• If pursuing the external trigger: Modal support ticket with window 2026-07-21 01:02–01:17 UTC, workspace apps above

Start in cloud_run_gateway.py at _call_modal_worker_dispatch and _run_modal_operation, then inspect the timeout plumbing and call_modal_worker_dispatch docstring in worker_dispatch.py. Use the supplied gateway and worker log queries to confirm the failure pattern. Done means the selected priority fixes address bounded dispatches, request-path failure handling, and unknown-path routing without regressing healthy Modal-primary traffic.