NIP-50 search pool uses READ_DATABASE_URL raw, bypassing all replica fencing — live in bb-block and bb-public today
- Dominant language
- Rust
- Stars
- 32.7k
- Forks
- 4.3k
- Avg merge
- 1d 13h
- Merged PRs (30d)
- 253
Description
## Summary
NIP-50 search creates its own eager, default-timeout SQLx pool directly from `READ_DATABASE_URL` (`crates/buzz-relay/src/main.rs:389-402` at `dd26caa9f`; same code at `origin/main` `ab55fee81`). This pool bypasses **all** of the replica fencing machinery in `buzz-db`: no heartbeat/freshness proof, no 150ms bounded acquire, no writer fallback, and `BUZZ_REPLICA_READ_MAX_AGE_MS` cannot gate it (the budget gates hydration only — the initial FTS query that produces the IDs runs unfenced).
**This is not a #3268 defect and it is live in production today:**
- Introduced by #2084 (`29c48883d`, 2026-07-18) — predates the #3268 branch.
- Present byte-for-byte in both deployed images: bb-block `sha-dd222a5`, bb-public `sha-22be8bb` (verified via `git show :crates/buzz-relay/src/main.rs`).
- Both production ESOs template `READ_DATABASE_URL` → the Aurora `cluster-ro` endpoint (bb-block: `externalsecret-buzz-secrets.yaml:28`, added in bb-block #487, survived #539; bb-public: equivalent line, `cluster-ro-cv206gsyeugr`). The rendered bb-block `deployment-buzz.yaml:98-103` wires it into the pod. Relay logs in prod will show `Search service ready ... replica=true`.
## Measured failure modes (Max's live Helm redteam at `dd26caa9f`, PG17 physical streaming; evidence: `RESEARCH/PR3268_LIVE_REPLICA_REDTEAM.md`, thread cf5deba4 in #buzz-read-only-replica-usage)
1. **Reader loss stalls live search ~30s** — NIP-50 request exceeded the 10s client deadline; SQLx pool timeout surfaced at ~30s instead of the fenced paths' 150ms writer fallback.
2. **Replica lag makes acknowledged writes unsearchable** — with WAL replay paused, search returned zero hits for a message the writer had just accepted, while the fenced channel paths correctly fell back `writer/stale`. Resuming replay fixed the identical query.
3. **Reader loss blocks pod startup/rollout** — the eager `.connect().await?` is fatal; with the standby absent a replacement pod stayed `0/1` past 50s while the fenced lazy pool correctly logged `serving all-writer`.
4. **B=0 does not disable replica FTS** — routed hydration reported `writer/disabled` while FTS still logged `replica=true` and queried the reader.
## Aurora nuance (why prod hasn't burned yet)
Max's rig used vanilla physical streaming, where a dead standby = failed connect. Aurora's `cluster-ro` endpoint resolves to the writer when no readers exist and re-points on failover, so modes 1 and 3 are softened to DNS-failover windows rather than hard outages. Mode 2 is not softened: observed bb-block replica lag is 5-25ms baseline but spiked to **3,976ms** at 2026-07-29 16:11Z (`aws.rds.aurora_replica_lag`, fine-grained query) — during any such window, freshly accepted messages are silently unsearchable. Frequency ~1-2 one-minute windows/day.
## Fix options
- **Minimal (recommended first):** pin the FTS pool to `database_url` unconditionally. One-line semantic change; search load returns to the writer (it lived there before #2084).
- **Proper (follow-up):** route the initial FTS query through the same bounded/fenced writer-fallback contract as the routed seams in #3268.
Interim config-only mitigation if search symptoms appear before the fix deploys: remove `READ_DATABASE_URL` from the ESO (Max verified this rollback rolls out cleanly).
## Verification gate (per Max — required before any future reader URL enables replica FTS)
1. Reader-down traffic latency (search must fall back within budget, not 30s).
2. Reader-down cold boot / scale-out (pod must become ready).
3. Paused-replay read-your-own-write search (accepted message must be findable).
Note: #3622 (CI enforcement of the buzz-db fixtures) would **not** catch this — it is a separate pool outside `buzz-db`.
Refs: #3268, #2084, #3622. Found by Max's live redteam; provenance verified by Eva.
Contributor guide
Assessment
This issue has not been assessed yet.