Distributed Redis DPoP-jti replay store — maintainer decisions (fail-closed, failover hazard, HA, cost)

## Distributed Redis DPoP-`jti` replay store — maintainer decisions for steer

This issue records the design decisions made while implementing the **distributed Redis-backed DPoP-`jti`
replay store** (the horizontal-scaling enabler), per the "proceed on the best choice, document for
after-the-fact steer" rule. Branch: `feat/redis-replay-store` (draft PR to follow). The store lives in
`src/redis_replay.rs` behind the opt-in `redis-replay` Cargo feature; it's selected at runtime by
`SOLID_SERVER_REPLAY_REDIS_URL`. **Default build/tests/conformance are byte-identical** (feature off →
the existing in-memory store; conformance stays 41/41).

### Why it exists
The verifier's default replay store is **per-instance** (an in-memory set). Correct for one node, but the
moment the server runs horizontally behind a load balancer, a `jti` consumed on instance A is invisible to
instance B — so a captured DPoP proof can be replayed against a *different* instance within its freshness
window. The in-memory store also fails closed at a bounded capacity (a single-instance safety bound, not a
scaling story). A **shared** Redis set (`SET dpop:jti: 1 NX PX ` — the `NX` reply IS the
New/Replay signal in one atomic round-trip) fixes both: every instance marks into the one set.

---

### Decision 1 — Redis-outage failure mode: **FAIL-CLOSED** (chosen; non-negotiable)
On ANY Redis error (unreachable, connect/op timeout, command failure, malformed reply) the store returns
`ReplayBackendError`, which the verifier maps to its existing **503** path (`replay_fail_closed` defaults
true). We **never fail open**.

- **Rationale:** a fail-open Redis outage would be a **global replay-protection bypass across the whole
fleet** — strictly worse than rejecting traffic. Availability loss during a Redis outage is the correct
trade for a resource server holding personal data.
- An unreachable Redis additionally **fails the server at boot** (an eager PING during `connect`), so an
instance never runs with silently-disabled replay protection.
- This is **chosen and treated as non-negotiable** per the design. Flagging it here only so the maintainer
is aware availability is intentionally coupled to Redis availability (mitigated by HA — Decision 3).

### Decision 2 — the async-replication-on-failover hazard (**needs maintainer-approved acceptance**)
Redis primary→replica replication is **asynchronous** by default. A `SET NX` acked by the *old* primary but
**not yet replicated** to a replica is **LOST** when a Sentinel/Cluster failover promotes that replica.
After failover the same `jti` can be marked `New` a second time → a **bounded replay window**.

- **Bound:** the window is at most **one proof-freshness window** (the DPoP `iat` max-age + clock
tolerance, the same TTL we `PX`), and only for proofs in flight across the failover instant. It is not an
unbounded bypass — a genuinely stale proof is still independently rejected by the proof's own `iat`
freshness check.
- **Options for the maintainer to choose between:**
1. **Accept the bounded window** (recommended default): simplest, lowest latency; the exposure is small
and time-bounded, and failovers are rare.
2. **`WAIT` for replica ack** after each `SET NX` (e.g. `WAIT 1 `): closes most of the window at the
cost of a per-mark latency hit and reduced availability if a replica is down.
3. **A CP store** (e.g. Redis with RedisRaft, or a different CP backend behind the same `ReplayStore`
seam): strongest guarantee, highest operational cost.
- **Proceeding with (1)** (accept the bounded window) as the documented default, since the trait is
backend-agnostic and (2)/(3) can be added later without touching the verifier. **Please confirm or
redirect.**

### Decision 3 — HA topology
- **Same-AZ Redis** (or same-AZ primary) to keep the per-mark RTT low (it's on the auth hot path; a tight
~50 ms op timeout is enforced).
- **Sentinel or Cluster** for failover. Because `jti` is high-entropy and uniformly distributed, **sharding
by key** (Redis Cluster hash-slots) scales writes linearly with no hot-spotting — the keyspace is a flat
set of independent keys with per-key TTLs, ideal for sharding.
- Each app instance points at the same logical Redis endpoint (Sentinel/Cluster client or a proxy).

### Decision 4 — cost
- One small Redis (or a managed Redis HA pair) per region. Memory is bounded by the live-`jti` set =
(authenticated RPS × proof-TTL-seconds) keys, each tiny (`dpop:jti:` → `1`, auto-expired by `PX`).
At, say, 1000 authed RPS and a 60 s TTL that's ~60k keys (single-digit MB) — a small instance suffices.
Managed Redis HA is a modest fixed monthly cost; self-hosted is a small same-AZ node pair.

---

### Implementation notes (for reviewers)
- **Off the async runtime:** `ReplayStore::mark` is a SYNC trait method called from inside the axum Tokio
runtime. Redis I/O runs on a **dedicated OS thread** owning an **r2d2 pool of blocking connections**
(mirroring the verifier's `net.rs` off-runtime pattern); `mark` ships a job over a channel and blocks on a
plain `std::sync::mpsc` reply (not a runtime entry) — so no Tokio worker is ever parked on the Redis RTT,
and there's no "runtime within a runtime" hazard.
- **Tight timeout:** a ~50 ms per-op socket read/write timeout (the hot-path bound) so a slow Redis becomes
a fast 503; a more generous (but still bounded) pool-acquisition timeout absorbs cold-connect + r2d2
scheduling without weakening the hot-path bound.
- **Full `jti` as the key** (namespaced, never hashed/truncated): a hash collision would be a false replay
or — worse — a missed replay.
- **One concrete replay type:** a `BackendReplay` enum (`InMemory` | `Redis`) keeps `main.rs` on a single
monomorphised `SharedReplay` regardless of backend, so the verifier/cache/AppState/router
wiring is unchanged.
- **Tests:** unit tests + an integration test proving (a) cross-instance replay rejection (two stores, one
Redis), (b) fail-closed at boot AND at mark-time (a self-contained fake-Redis test + an unreachable-port
test), and (c) TTL expiry. Security-critical → exhaustive + roborev + adversarial verify before merge.

🤖 PSS agent — @jeswr's agent for `prod-solid-server` / the Solid app+Pod-Manager suite

Start with src/redis_replay.rs and the redis-replay feature wiring, then review the fail-closed verifier path and the implementation notes in this issue. The work is done when the maintainer resolves the failover decision and the unit and integration tests cover cross-instance replay rejection, boot and mark-time failures, and TTL expiry.

Written by the indexing model from the issue text.