Production-readiness gap analysis (meta / tracking)

## Production-readiness gap analysis

This is a tracking meta-issue for the gap between the current service and a hardened, internet-facing production deployment. It was produced by a full-codebase audit across four dimensions (security/abuse, data durability, operations/lifecycle, and API/protocol robustness) against `main` (commit `8fd09ee`).

### Verdict

The service is **not production ready**. The code is genuinely well-built at the unit level — clean transactions, `STRICT` tables, a sound monotonic-`seq` cursor design, good regression tests, privacy-conscious tracing — but it was built as a functional prototype. Everything that separates "works on my machine" from "safe on the open internet" is missing. Issue #112 (in-memory DB default) is one symptom of this broader pattern.

### The gating question — decide this first

**No authentication or authorization on any RPC. Anyone can call SendNote, FetchNotes, StreamNotes, Stats. This needs to be an explicit product decision (open relay by design, or gated), because everything below depends on the answer.**

If the answer is "open relay by design", then rate limiting, storage quotas, and note validation become mandatory compensating controls. If "gated", an auth interceptor changes the shape of several of the sub-issues below. Nothing else should be finalized until this is settled.

Update (design discussion, #115): a third framing — economic gating via inclusion-proof validation (#116). If the NTL only accepts a note once its commitment is registered on-chain, the on-chain fees already paid to create the note bound write-flooding and storage growth, largely subsuming the write-side of #117/#118 and closing the squatting vector (#47). Under this framing, any remaining auth question narrows to the read path (FetchNotes/StreamNotes/Stats) only. See #115 for the full breakdown.

### What is already solid (so we scope realistically)

SQL injection is not possible (parameterized diesel throughout); note size and tag-count are capped; the cursor/pagination design is correct and well-tested; a gRPC health service and reflection exist; tracing is good; and release/CI discipline (CHANGELOG, SemVer, multi-arch images, cargo-deny) is above average. The foundation is good; the hardening layer is absent.

### Gaps by theme

Security and abuse resistance (weakest area):
- No auth/authz on any RPC (the gating question above).
- No note validation — `SendNote` stores arbitrary blobs; no on-chain commitment check, no header/details binding.
- No rate limiting of any kind.
- Unbounded storage — no per-sender/per-tag/global quotas; disk-full takes the service down.
- No TLS (plaintext gRPC only).
- Responses are row-count-capped, not byte-capped — ~256 MB amplification and permanent client wedge.

Streaming path (fragile, DoS-prone):
- Silent note loss for connected subscribers (cursor advances before delivery; delivery is waker-gated).
- Unbounded subscriptions, O(tags) sequential polling, control-flood starvation, per-subscriber batch cloning, subscription leaks, full-backlog replay on resubscribe.

Durability (beyond #112/#111):
- Fail-open DB creation — a wrong/unmounted path silently starts an empty database (a second trigger for the #101 dead band).
- Migrations run implicitly per pool-connection against the live DB.
- Retention is one unbounded DELETE; no explicit durability PRAGMAs; no backup/restore story at all.

Operations and observability:
- No graceful shutdown (SIGTERM drops in-flight work, streams, buffered telemetry).
- Server errors exit with code 0 — restart-on-failure never triggers.
- Health reports SERVING unconditionally regardless of DB state.
- Request-duration metric double-counts every request; no error-rate or saturation metrics.
- Docs/CHANGELOG reference telemetry env vars and flags the code doesn't read.

### Relationship to existing issues

These flesh out existing placeholders rather than duplicating them: #44 (spam) is the umbrella for rate limiting + storage quotas; #47 (squatting) is the header front-running / first-writer-wins denial; #96 (stream ignores cursor) overlaps the streaming work; #101 is the sequence-reset dead band that fail-open DB creation feeds; #111 (Postgres) is the durability/HA direction; #95 is the maintenance busy-spin; #75 is load testing.

### Sub-issues

Grouped by severity.

Critical:
- [ ] #115 — Define and enforce a trust/auth model for all RPCs (the gating question)
- [ ] #116 — SendNote accepts arbitrary unverified blobs (no on-chain commitment / header-details binding)
- [ ] #117 — No rate limiting or per-client request quotas
- [ ] #118 — Unbounded storage growth — no quotas or disk-full handling
- [ ] #119 — No graceful shutdown (SIGTERM drops in-flight work, streams, telemetry)

High:
- [ ] #120 — No TLS on the gRPC server; missing HTTP/2 connection hardening
- [ ] #121 — Responses/streams are row-count-capped, not byte-capped — amplification and client wedge
- [ ] #122 — StreamNotes can silently drop notes for connected subscribers
- [ ] #123 — StreamNotes resource/DoS hardening (subscriptions, polling, cloning, leaks)
- [ ] #124 — Fail-open database creation silently starts an empty DB
- [ ] #125 — No backup/restore story for the note database
- [ ] #126 — Server errors exit with code 0 — restart-on-failure never triggers
- [ ] #127 — Health service reports SERVING unconditionally regardless of DB state
- [ ] #128 — RequestTimer double-counts requests; no error-rate or saturation metrics
- [ ] #129 — SendNote is not idempotent — duplicate send returns INTERNAL
- [ ] #130 — Legacy-cursor reset never converges (rcursor computed from original cursor)

Medium / low:
- [ ] #131 — Migrations run implicitly per pool-connection against the live DB
- [ ] #132 — Retention cleanup: batch the DELETE and set explicit durability PRAGMAs
- [ ] #133 — Container runs as root, no HEALTHCHECK; compose uses :latest + hardcoded Grafana password
- [ ] #134 — Docs and CHANGELOG reference telemetry env vars/flags the code doesn't read
- [ ] #135 — Proto package lacks a version suffix; no buf breaking check in CI
- [ ] #136 — Production-readiness papercuts (bundle)