block / block/buzz

[bug] NIP-43 maintenance sweep enumerates tombstoned communities, logging "community write fenced" every ~60s forever after a completed deletion

Open
#7,558 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

**Describe the bug**

After a whole-community deletion completes successfully via `buzz-admin deletions`, the relay's periodic NIP-43 membership reconciliation sweep keeps enumerating the deleted community's tombstone row and attempting a guarded write against it. The community's own deletion fence refuses the write, and the relay logs a `WARN` once per maintenance cycle — roughly every 60 seconds — indefinitely.

The deletion itself is fine. Every checkpoint completed, all fenced tables were purged, and `verify_cross_store_absence` passed. The row simply remains in `communities` as `deletion_state = 'tombstone'` while the request sits at stage `retention_pending`, awaiting physical retention expiry — which is by design. The sweep just doesn't know to skip it.

Root cause looks like the community enumeration helper, which selects every row with no `deletion_state` predicate:

```rust
// crates/buzz-db/src/store/usage.rs:399 (identical on main @ 051c3a27)
let rows = sqlx::query_as::<_, (Uuid, String)>("SELECT id, host FROM communities")
```

Both `usage_community_hosts()` (Maintenance) and `bootstrap_community_hosts()` (Bootstrap) route through `community_hosts_with_operation`, so `reconcile_nip43_membership_snapshots_with_purpose` in `crates/buzz-relay/src/handlers/side_effects.rs` receives tombstoned communities and hands each to `publish_nip43_membership_list`, which the fence then rejects.

**Steps to reproduce**

1. Have two communities on a relay, one stranded/empty (ours came from a `RELAY_URL` authority change — `minime.local:3000` vs `minime:3000`).
2. Delete the stranded one through the supported control plane: `buzz-admin deletions submit`, then `approve`, then `run`. Confirm it reaches a completed state with all checkpoints `completed`.
3. Leave the relay running and watch the logs: `docker logs --tail 2000 | grep "NIP-43 membership reconciliation failed"`.
4. A `WARN` appears every ~60s, forever. Restarting the relay does not clear it — the row is still in `communities`.

**Expected behavior**

The maintenance sweep should only reconcile communities that are actually serving — i.e. filter to `deletion_state = 'active'` (excluding `quiescing`, `fenced`, and `tombstone`), either in `community_hosts_with_operation` or in the sweep itself. A community that has been deliberately deleted and fenced should not be a reconciliation target, and failing to write to it is expected rather than exceptional.

**Version and platform**

- Buzz version: relay at commit `3c7f288c60d67df78577b237e27c3dfc8831aaa1`, image `ghcr.io/block/buzz@sha256:919280d68dc2b61d91d92136e9ab19566c40b6e1afc1937df8986d73abe2de2c`. Verified still present on `main` @ `051c3a270be9` by reading both files — no fix in between.
- OS: macOS 26.6.2 (Apple Silicon), Docker inside a Colima VM. Relay deployed from `deploy/compose`.

**Logs / additional context**

```json
{"timestamp":"2026-09-10T14:01:29.922464Z","level":"WARN","message":"NIP-43 membership reconciliation failed","community_id":"43b71a51-c183-40c9-a6e1-7f223271f2c7","host":"minime.local:3000","error":"database error: error returned from database: community write fenced: community 43b71a51-c183-40c9-a6e1-7f223271f2c7 generation 1","target":"buzz_relay::handlers::side_effects"}
```

Cadence measured over the last 2,000 log lines: 278 occurrences, inter-arrival gaps min 59.6s / max 60.0s / avg 59.9s. First occurrence `2026-09-06T03:34:36Z`, which is 59 seconds after the deletion committed at `2026-09-06T03:33:37Z`.

Community rows:

```
id | host | deletion_state | gen | deleted_at
--------------------------------------+-------------------+----------------+-----+-------------------------------
43b71a51-c183-40c9-a6e1-7f223271f2c7 | minime.local:3000 | tombstone | 1 | 2026-09-06 03:33:37.066095+00
0c170e88-e9b7-451f-ba33-137131e71318 | minime:3000 | active | 0 |
```

**Impact** is limited but not zero: unbounded WARN noise in relay logs, a monotonically climbing `buzz_nip43_membership_reconciliation_failures_total`, and — the reason I'd argue it's worth fixing — it permanently poisons that metric and log channel as a signal, so a *genuine* reconciliation failure on a live community is much harder to notice. The active community is unaffected; health checks, NIP-42 auth and normal client traffic are all fine throughout.

This appears distinct from #5433, which concerns reconciliation failing for a *live* community after events are deleted (causing channel-creation 403s). This one is about a deliberately deleted community that should no longer be swept at all.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.