block / block/buzz

relay: post-OOM silent permanent crashloop — pre-bind fleet-wide NIP-43 sweep exceeds 120s startup probe at 42k communities

Open
#4,555 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## Summary

After an OOMKill (see the companion "Blob" memory issue), bb-public relay pods enter a **silent permanent crashloop**: 22–35MB RSS, port 8080 never binds, zero log lines at `RUST_LOG=error`, startup probe SIGKILL every 120s, forever. Observed restart counts: 142 / 205 / 52. Node problem ruled out.

## Root cause (confirmed locally against prod image sha-c104eec)

`main.rs` awaits `handlers::side_effects::reconcile_nip43_membership_snapshots(&state)` **before** `serve()` binds the listener. The sweep:

- loads ALL communities (`SELECT id, host FROM communities`, buzz-db `usage.rs:347`),
- loops them **sequentially**, 2 queries per community, plus a snapshot publication per repair.

Measured on an isolated docker rig (prod image, prod-fidelity env): 1001 communities seeded, all snapshot events deleted → **~9.5ms/community** repair cost, ~9.5s of publishes logged before `buzz-relay TCP listening`; clean-check pass boots in 11s.

At bb-public scale (**~42k communities**) that extrapolates to **6–8 min check-only**, worse with repairs — vs the **120s** startup probe. The probe kills the pod mid-sweep; the restart starts over at community #1. Permanent crashloop. Matches the prod `pg_stat_activity` signature (cycling events/relay_members queries, advisory locks). The "no logs" presentation is `RUST_LOG=error`: a healthy boot at that level logs nothing.

## Fix

- Emergency fix: **PR #4554** — pre-bind reconciles only the deployment community; fleet sweep moves post-bind, jittered + leader-gated (session advisory lock), with sweep start/progress/complete/fail telemetry.
- Follow-up (separate PR, agreed scope): pagination/resume for the sweep + per-query lock/statement deadlines.

## Evidence locker

Buzz channel `buzz-the-blob` (8c53b83d-2692-4621-a5a4-a00b826f746d), thread 62082b57 — local measurements (Eva), source trace (Wren), prod telemetry (Sami).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.