relay: post-OOM silent permanent crashloop — pre-bind fleet-wide NIP-43 sweep exceeds 120s startup probe at 42k communities
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
After an OOMKill (see the companion "Blob" memory issue), bb-public relay pods enter a **silent permanent crashloop**: 22–35MB RSS, port 8080 never binds, zero log lines at `RUST_LOG=error`, startup probe SIGKILL every 120s, forever. Observed restart counts: 142 / 205 / 52. Node problem ruled out.
## Root cause (confirmed locally against prod image sha-c104eec)
`main.rs` awaits `handlers::side_effects::reconcile_nip43_membership_snapshots(&state)` **before** `serve()` binds the listener. The sweep:
- loads ALL communities (`SELECT id, host FROM communities`, buzz-db `usage.rs:347`),
- loops them **sequentially**, 2 queries per community, plus a snapshot publication per repair.
Measured on an isolated docker rig (prod image, prod-fidelity env): 1001 communities seeded, all snapshot events deleted → **~9.5ms/community** repair cost, ~9.5s of publishes logged before `buzz-relay TCP listening`; clean-check pass boots in 11s.
At bb-public scale (**~42k communities**) that extrapolates to **6–8 min check-only**, worse with repairs — vs the **120s** startup probe. The probe kills the pod mid-sweep; the restart starts over at community #1. Permanent crashloop. Matches the prod `pg_stat_activity` signature (cycling events/relay_members queries, advisory locks). The "no logs" presentation is `RUST_LOG=error`: a healthy boot at that level logs nothing.
## Fix
- Emergency fix: **PR #4554** — pre-bind reconciles only the deployment community; fleet sweep moves post-bind, jittered + leader-gated (session advisory lock), with sweep start/progress/complete/fail telemetry.
- Follow-up (separate PR, agreed scope): pagination/resume for the sweep + per-query lock/statement deadlines.
## Evidence locker
Buzz channel `buzz-the-blob` (8c53b83d-2692-4621-a5a4-a00b826f746d), thread 62082b57 — local measurements (Eva), source trace (Wren), prod telemetry (Sami).
Contributor guide
Assessment
This issue has not been assessed yet.