relay: per-pod anon memory staircase to 10Gi OOMKill every 24-48h — migrating singleton 'carrier' role, ~173x anon-per-wire-byte amplification

## Summary ("The Blob")

bb-public relay pods show a per-pod anon-memory staircase ending in OOMKill at the 10Gi limit every 24–48h. Each OOM drops all WS clients and stampedes DB re-pooling; on 8/3 this produced ~1,060 user-visible pool-timeout errors. Image at first filing: ghcr.io/block/buzz:sha-c104eec; fleet now on sha-be95a8a (behavior unchanged).

## What telemetry establishes (Datadog, 07-31 → 08-03)

- **Not fleet-wide** — a migrating singleton "carrier" role: exactly one pod ramps at a time (+10–15 GB/h anon), mutual exclusion held over 36h, and within ~1 min of killing the carrier the role hands off to another pod.
- **Same-minute network association, 10/10 carriers**: pod tx steps +5–41% vs same-timestamp fleet median at the exact onset minute (negative control: 123 random split points, null p90 = 1.054 — nine of ten carrier steps exceed it).
- **~173x median anon-per-excess-wire-byte amplification** (range 68x–1007x): the carrier absorbs ~200 MB/min of anon while carrying only ~1–2 MB/min more traffic than peers. This kills any hypothesis where retained bytes track wire bytes, and makes the carrier invisible to volume thresholds.
- **Causally implicated (local intervention, 2026-08-03): the 300s usage-poller leader's per-community metrics path.** At 45k communities on matched containers (exact head `e8e258170`), the usage-poller **leader** under `BUZZ_USAGE_METRICS_PER_COMMUNITY=all` retains ~458 MiB (peer flat ~10 MiB; `=off` leader flat) with **zero scrapes**; one ~97 MB /metrics render (1,125,200 `buzz_community*` series) adds ~0.6–0.8 GiB. First non-circular estimates: ~395 B/series retained (matched all−off), ~540–659 B/series render delta. The 60s NIP-43 sweep is **exonerated** as the retained-step source (transients only). Earlier sticky/slow-reader REQ hypothesis is superseded.
- **Open: a production-only amplifier.** Local plateaus at ~0.46–1.18 GiB; prod climbs 300–900 MB/min to 10 Gi with exposition at parity with the local rig (88.7 vs 85.8 B/series — the multiplier is not exposition size). The local concurrency probe (2026-08-03, exact head `e8e258170`, warm leader at 45k communities) split the candidate: **concurrent-render overlap is confirmed and large** — four-way overlap added ~2.22 GiB anon (441 → 2,658 MiB, cgroup peak 2,835; no per-render quotient — concurrent scaling is not established as linear) and left a **post-client-exit anon high-water** of ~2,284 MiB through t+184s (all four client processes gone by t+15; server-side render completion is not instrumented, and this round's HTTP receipts were lost to a later overwrite — see rig evidence notes) — while **abandonment-specific retention is falsified in this local shape**: 4 renders abandoned client-side at 100 ms (DD-timeout analog; `spawn_blocking` is unabortable — metrics-exporter-prometheus 0.18.3 `http_listener.rs:157`, tokio 1.52.3 `blocking.rs:108`; accept loop unbounded, `http_listener.rs:69-82`) added no detectable incremental durable anon plateau in that warmed arm (returned to baseline within ~3 s). **Production overlap status (updated after Agent-source verification, DataDog/datadog-agent @ 7.71.2 — the pinned prod image, `daemonset-datadog-agent.yaml:178` @ `2f619c845`):** the searched bb-public config shows exactly one declared `:9102` reader (pod-annotation openmetrics check, node agent — no `cluster_check: true`; DaemonSet provider `endpointschecks`, clusterchecks Deployment separate), istio excludes inbound 9102, and no ServiceMonitor/PodMonitor exists in-repo (scoped negative about that config search). **Agent self-overlap/retry is refuted from source**: a runner-global tracker gate skips a due run while the same check ID is in flight (`worker.go:150-152`; `tracker.go:50-60`; single tracker shared across the worker pool, `runner.go:51,68`), the ID is a stable FNV-64 hash with no time/nonce (`id.go:22-37`), and the entry is released only after `check.Run()` returns (`worker.go:205`). The relay check — the only openmetrics check in the repo with no `min_collection_interval` (others pin an explicit 30s floor) — runs on the **15s** agent default (`defaults.go:15`, applied on both loader paths: `python/check.go:84,275-276`, `checkbase.go:58,110-111`), so the deployed `"timeout": 20` (values.yaml:170) exceeds the check's own scheduling interval. The scheduler tick is unconditional (`job.go:181-220`, 15 one-second buckets per `job.go:86-95`), so behavior is **regime-dependent on server render duration D**: D<15s normal; 15–20s due-instants suppressed but runs complete (no abandonment); 20–30s every run abandoned client-side at 20s, requests every 30s, no sustained overlap; **D>30s: the 20s abandonment releases the tracker gate while the server render, once started, cannot be cancelled by client disconnect (`spawn_blocking` unabortability, tokio source cited above), and a fresh request fires every 30s — sustained ~D/30 concurrent server renders.** I.e., agent-side serialization does not serialize server-side renders once timeout < render duration; the cadence pathology and the overlap mechanism are not independent candidates — for D>30s the deployed config *manufactures* overlap (D≈120s reproduces the rig's four-way shape). This is a **conditional candidate — the strongest live amplifier candidate — with one premise left**: premise (b) is now **resolved from source** (DataDog/integrations-core @ 7.71.2, tag object `453b18b1`, commit `8074918686`): the check's `timeout: 20` is a requests `(connect, read)` tuple (`utils/http.py:296,390`) — a per-read inactivity timeout, not a total cap — but the relay's exporter fully buffers (`spawn_blocking(render)` awaited before any response byte, metrics-exporter-prometheus 0.18.3 `http_listener.rs:157-159`), so for D>20s the 20s read timeout is spent waiting for the first byte and the client raises ReadTimeout at ~20s. `check.Run()` returns in the agent at that point, releasing the tracker gate — no dependency on the server observing the disconnect. No HTTP-level retry exists at this tag (`utils/http.py` mounts no urllib3 Retry adapter; urllib3 imported only for `InsecureRequestWarning`, `:21`), so overlap count is governed by D/30, not retries. Remaining premise: **(a) prod leader render duration >30s — unmeasured**, plus a residual scope caveat (v2 scraper path verified; runtime v1-vs-v2 resolution not confirmed, though full buffering makes the streaming machinery moot either way). Skipped runs are **invisible as deployed** — the skip branch is a bare Debug+continue with no counter (`expvars.go` has no skip metric) and `DD_LOG_LEVEL=WARN` (`daemonset-datadog-agent.yaml:102-103`) suppresses Debug — so agent-side signals are check-run duration/error telemetry plus a run-count deficit against expected due instants (deficit alone establishes only boundary-crossing, not abandonment). **Instrument limits:** agent duration/error telemetry tests whether the 20s timeout branch is active (durations plateauing at ~20s with timeout errors), but it is right-censored at the timeout — it cannot distinguish a 21s server render from a 60s one, so it cannot establish premise (a). Server render duration D beyond the timeout needs a server-side signal: relay-side render start/finish/in-flight instrumentation or logs, an existing `:9102` request-duration signal that adds no request, or labeled indirect evidence (e.g., periodic memory/CPU signatures aligned to the agent's retry cadence). One instrument does not settle both premises. Do **not** measure by manually scraping a live carrier — a concurrent render costs GiB-scale live anon on a pod near its limit. Repeated-round staircase behavior remains untested locally (rig stopped at its disk floor); component attribution within a render (snapshot vs temporaries vs allocator) is still unmeasured.
- Ruled out locally: git pack-cache traffic (276+ clone soak, prod-fidelity env: cache plateaus at cap as reclaimable file pages; anon oscillates, no cumulative slope).

## Related but distinct finding (metrics, not memory)

Fleet-simultaneous gauge-darkness waves: on pods whose per-community series count crosses ~(1828, 2760) per family, `/metrics` output loses gauges/histograms in **render order** (counters survive). The earlier `max_returned_metrics` guess was retracted after reading the deployed check config. Related closed residual: `buzz_usage_poller_is_leader` reading a present non-null **0** on a ramping carrier is a starvation artifact, not a lease migration — the gauge is set only *after* `run_usage_metrics_tick` returns with `MissedTickBehavior::Skip` (`crates/buzz-relay/src/main.rs:1067` @ deployed `be95a8a`; same code at `:1157` @ PR head `e8e258170`), and every `set()` bumps the recency generation (metrics-util 0.20.4 `recency.rs:84-85`) so a re-set 0 never idles out.

## Canary experiment & interpretation (pre-registered, 2026-08-03)

Vehicle: [squareup/bb-public#347](https://github.com/squareup/bb-public/pull/347) — explicit control `PER_COMMUNITY: "all"` + `BUZZ_STORAGE_METRICS: "off"`; treatment is the one-line `all → off` flip (storage identically off in both arms). Expected treatment outcome, pre-registered by the local result: flat leader slope under `off`.

⚠️ **Interpretation limits, in writing before anyone reads the result:**
- **A flat treatment confirms the family, not the member.** `off` removes the fuel for *both* recorder retention and render/stacking work, so a flat leader cannot apportion between them (and does not kill the amplifier question). The member-separating local probe has since run (see amplifier bullet above): overlap confirmed as the multiplier locally but unestablished in production; abandonment-specific retention falsified locally.
- **A non-flat treatment does not rule out the per-community family.** Interpret the slope/level delta, not a binary flat/non-flat result: flatter or lower means the family contributed; fully flat means it was sufficient to explain the observed ramp under this deployment; unchanged within uncertainty means no detectable contribution here and surviving paths should be prioritized; reduced but still rising means multiple contributors. All 12 GROUP-BY collections remain active in both arms.

## Secondary env bug (found during rig build, worth its own small fix)

Prod sets `BUZZ_GIT_MAX_PACK_BYTES="5.24288e+08"` — scientific notation fails the u64 parse and **silently** falls back to the 500MiB default. Config parse failures of explicitly-set env vars should at least log.

## Evidence locker

Buzz channel `buzz-the-blob` (8c53b83d-2692-4621-a5a4-a00b826f746d), thread 62082b57 — full receipts: Datadog queries/tables (Sami), source analysis (Wren), local soak rig (Eva). Rig: `.scratch/blob-warroom/` (compose project buzz-blob).