[Epic] Replication reliability & scale — foundations, adaptive topology, and locking

## Epic: Replication reliability & scale — foundations, adaptive topology, and locking

This epic organizes a multi-release plan for Harper replication into tracked workstreams. It comes out of a deep architecture review of `replication/` and the core substrate it depends on (full review notes available on request). The goal: turn a long tail of hard-won reliability fixes into a sound foundation that scales across topologies (full-mesh, hub-and-spoke/transitive, sharded, WAN, many-node) and supports adaptive behavior and distributed locking as *additive* features rather than new complexity on cracks.

**Storage assumption:** RocksDB is the canonical (and going-forward only) replicated engine. LMDB is being deprecated — **this plan does not design around LMDB**. That simplifies the keystone work below, because the RocksDB transaction log is already partitioned per origin.

### The core finding

Reliability has been hard-won because a few foundational design choices each radiate a whole *family* of bugs, and the monolithic protocol engine (`replicationConnection.ts`, ~4,400 lines and growing — up ~800 lines since this epic was filed; `replicateOverWS` is a single ~3,250-line closure) makes every fix high-risk. **Five of our strategic concerns are blocked on, or dramatically de-risked by, two foundations.** Land these first and adaptive routing, dedicated threads, robust sharding, and locking become straightforward.

**Foundation 1 — A single source of truth for connection & health state.** Today the main-thread orchestrator keeps an *edge-triggered, inferred mirror* of connection state; the real sockets live on worker threads; live metrics (latency, back-pressure) live in shared-memory buffers the main thread never reads. That split is the direct cause of the connection-truth bug class (#289, #349, #357, #233) **and** is exactly the missing bridge that adaptive replication (#218) needs. Fix it once → retire a bug class *and* unlock adaptive routing + adaptive threading.

**Foundation 2 — Per-origin transaction-log convergence.** Provably-correct transitive replication (#399), per-originating-node connections (#193), and per-origin observability (#192) all require resume cursors that track *each origin's own position*. The RocksDB transaction log already partitions per origin (`RocksTransactionLogStore` `nodeLogs[]`/`logById`); the work is to make replication's cursor layer consume per-origin positions as the *primary* mechanism and carry them transitively across hops. This is the keystone for transitive correctness and topology scale.

### Root-cause themes (each generates multiple bugs)

- **A. No single source of truth for connection/health** — the orchestrator's `connected` bit desyncs whenever a terminal/idle state is reached without the expected transition event; back-pressure/latency sit in shared buffers the main thread never reads. → #289, #233, #357, #349, #217
- **B. The resume cursor is *trusted*, not *checked*** — narrow error classification over-holds (a permanently bad source pins the cursor forever) or silently gaps (a `[T, head]` hole is never detected because live tail traffic keeps `connected:true`). No receive-side sequence-gap detection. → #426, #429, #385, #386, #388
- **C. Inconsistent backoff; one-shot intents leak into persistent listeners** — subscription-setup has no backoff (spins the main thread); `forceResubscribe` binds to a persistent listener; expensive per-reconnect work (TLS context creation) amplifies into OOM. → #327, #349, #288
- **D. Per-origin vs. interleaved cursor** — transitive flooding happens because a relay re-streams an already-applied tail (the relayed cursor is the *proxy's* position, not the *origin's*). Fixed by Foundation 2. → #399, #193, #192, #197
- **E. Residency is overloaded** — `replicateTo` (transient per-request routing) is laundered into the same persistent `residencyId` as durable shard policy, triggering invalidation broadcasts to never-target nodes. → #208, #211, #212, plus the sharding crash family #197–201, #257
- **F. Monolithic code blocks safe change** — a ~3,250-line closure with ~50 interdependent locals, no protocol version negotiation, and an inbound decode loop that *swallows errors* (silent partial-message loss).
- **G. Wall-clock LWW makes clock skew a correctness property** — concurrent full-record writes resolve by node name and silently discard the loser. This is the gap exclusive locking (harper#483) closes for explicitly-serialized keys.
- **H. Sync consumers of MaybePromise storage reads** *(added 2026-07-01)* — RocksDB `store.get()` returns a Promise on a block-cache miss, so sync consumers of system-table point reads (handshake resume-cursor reads, `hdb_nodes` lookups) work on warm caches and small tables, then silently break as tables grow or caches go cold — the definitive root cause of the 5.1.x post-upgrade stall family. Fixed pointwise by `getSync` conversions (#476, #484) and by typing the `__dbis__` store so sync `get()` misuse is a compile error (#485). The residual — typed `Store` for the remaining system stores + a lint rule — is tracked in W11 (#440). → #476, #484, #485, plus the superseded scan-fallback bandaids #461/harper#1463

### Workstreams

Foundations (do first):
- **W1** (#431) — Connection & health: single source of truth *(Foundation 1)*
- **W2** (#432) — Cursor correctness & divergence detection
- **W3** (#433) — Reconnect/storm control & resource bounds
- **W4** (#434) — Per-origin transaction-log convergence *(Foundation 2 / keystone)*

Features & cross-cutting:
- **W5** (#218) — Adaptive replication routing *(existing issue, adopted as a child of this epic)*
- **W6** (#435) — Adaptive / dedicated replication threads
- **W7** (#436) — Robust sharding (residency-vs-routing split)
- **W8** (#437) — Observability & metrics pipeline
- **W9** (#438) — Exclusive distributed locking (replication integration) *(core API in harper#483)*
- **W10** (#439) — Performance
- **W11** (#440) — Code organization, protocol versioning & decode-loop safety *(enabler)*
- **W12** (#444) — Hierarchical aggregation / re-origin relay mode for large fan-in trees *(distinct mode from W4; design-spike-gated; opportunity-driven)*
- **W13** (#510) — Base-copy & catch-up path *(added 2026-07-01 — the #480 family made the copy path the highest-churn area; it needs a single owner rather than pieces scattered across W2/W3/W10)*

### Proposed sequencing

- **Phase 1 — Foundations & safety:** W1 (#431), W2 (#432), W3 (#433), W11 early steps (#440 — decode-loop fix, protocol version negotiation, FrameWriter), W8 Tier 1 (#437 — cluster_status quick wins), W13 correctness items (#510 — #495/#489 are already in flight).
- **Phase 2 — Keystone & observability backbone:** W4 (#434), W8 Tier 2 (#437 — metrics + divergence detection), W11 continued (#440), W13 watchdog consolidation (#510, once W1 lands).
- **Phase 3 — Adaptive behavior & robust sharding:** W5 (#218), W6 static pool (#435), W7 (#436), W10 incremental (#439), W13 copy-transport perf (#510 — direct binary relay).
- **Phase 4 — Locking & advanced scale:** W9 (#438), W6 elastic (#435, if justified), W10 structural (#439), large-cluster validation (#263, #300). **W12 (#444)** re-origin relay is design-spike-gated and opportunity-driven — may pull earlier than this phase.

W9 (locking) depends only on the core metadata/audit wiring, not on W1/W4, so it can start in parallel once the metadata bit lands.

### Shipped since filing (ledger, updated 2026-07-01)

Point fixes landed since this epic was filed, mapped to the theme they belong to. They confirm the theme taxonomy — every one falls into a predicted family — but none retires a theme; the workstreams do that.

- **Theme A (connection truth):** #233 closed; #420 open-but-idle watchdog (#424); #463 reconcile-level fallback for connected:true copy stalls; #466 pause-stall watchdog + wedge re-drive + never-connected backstop; #471 empty-subscription delayed-close fix (#475). **Note the shape of these fixes: they are now ~6 layered edge-triggered watchdog/fallback recovery mechanisms** (~97 watchdog references in `replicationConnection.ts`), and they have already produced their first *interaction* bug (the #466 false-positive force-reconnect). The risk has shifted from "wedges with no recovery" to "overlapping recovery layers with subtle interactions" — which sharpens, not weakens, the case for W1: its acceptance criterion now includes demoting these watchdogs to telemetry/assertions (see #431).
- **Theme B (cursor trust):** blob error classification broadened well past ENOENT — #429/#443 statusCode-forwarded permanent classification, #403/#405 ENOENT advance-past, the core blob-write taxonomy (harper#1480: PENDING_TYPE stamp, 503-transient vs 500-permanent), and the #388 proactive blob-repair sweep (`blobRepair.ts`). #426 closed via #428 (cursorless-start full copy) + the fast-skip; its send-side stale-cursor residual is explicitly owned by W2/W4. **Still missing: the bounded-retry escalation budget (a deliberately-transient 503 can still pin the cursor forever — observed live as the v4→v5 circular-503 wedge) and receive-side gap detection.** See the W2 status update (#432).
- **Theme C (storm control):** subscription-setup retries now have fixed delays in places, but #327/#349/#288 all remain open; the uniform backoff discipline is still the gap.
- **Theme F (monolith):** got *worse* — the file grew ~800 lines absorbing all of the above; the decode-loop error swallow and the absence of protocol versioning are unchanged. A `DESIGN.md` navigation guide and a proven low-risk decomposition pattern (extracted pure decision helpers + unit tests) emerged — W11 codifies it (#440).
- **Theme H:** discovered and mostly retired post-filing (see above); residual in W11.
- **Copy path:** the #480 family (copyApply durable snapshots, copy-flush pacer #483, copy-progress watchdog, control-plane-first ordering #422) plus new follow-ups (#489, #495) — consolidated into the new **W13 (#510)**.
- **Also shipped:** #506 directional controlled-flow enforcement on live connections (closes the #498 gap — directional route config was stored but not honored).

### Notes

- Existing granular issues stay where they are; each workstream cross-references the issues it retires. We can formally nest them under the workstreams later if useful.
- Two `from-jira` issues reference the abandoned NATS transport (#261 backlog counts, #266 large-clone payload). The *concerns* map onto the live WebSocket engine (audit backlog depth; oversized-message chunking through `MAX_PAYLOAD`); the NATS-specific issues should be reframed or closed.

🤖 Filed by Claude on behalf of Kris.