Replication W1: Connection & health — single source of truth

**Workstream W1 of #430 · Foundation 1 · the linchpin for reliability _and_ adaptive behavior**

## Status update (2026-07-01)

Since filing, the connection-truth gap has been patched by **layered edge-triggered recovery mechanisms** rather than a source of truth: the open-but-idle receive watchdog (#420 → #424), the copy-progress watchdog (issue #453), a separate copy-phase receive timeout, the pause-stall watchdog + wedge re-drive + never-connected backstop (#466), the reconcile-level fallback for connected:true copy stalls (#463), and the empty-subscription delayed-close repair (#471 → #475). That's ~6 overlapping watchdog/fallback layers (~97 watchdog references in `replicationConnection.ts`), and they've already produced their first **interaction bug** — the #466 false-positive force-reconnect in the wedge re-drive.

Two consequences for this workstream:
1. The current-state risk is no longer "wedges with no recovery" — it's "overlapping recovery layers with subtle interactions." Every new watchdog raises the cost of reasoning about the next one.
2. **New acceptance criterion:** once level-triggered truth exists, the point-fix watchdogs demote to telemetry/assertions — they should detect invariant violations and alert, not be load-bearing recovery paths. W13 (#510) owns the copy-side consolidation.

The underlying claims below still verify on main: back-pressure is still read only by `clusterStatus.ts` (`backPressurePercent`), and the per-(db,node) `worker.on('exit')` listener is still attached in `subscriptionManager.ts` (#357 open).

## Summary
Connection and health state has no single source of truth. The main-thread orchestrator (`subscriptionManager.ts` `connectionReplicationMap`) keeps an *edge-triggered, inferred mirror* of whether each peer is connected; the real sockets live on HTTP worker threads; live metrics (latency, back-pressure) live in shared-memory `Float64Array` buffers that the main thread never reads. These three views desync, which is the direct cause of a recurring class of connection-truth bugs — and it's the missing bridge that adaptive routing (#218) needs.

## Root cause / current state
- `connected` is set only by edge-triggered IPC (`connected-to-node`/`disconnected-from-node`). It desyncs whenever a terminal/idle state is reached without firing the expected transition: a silently-dead socket never emits `'close'` (#233/#420); a worker-load race arms a permanent "no subscriptions" close that pins `connected:false` even though the peer is reachable (#289).
- Back-pressure is written by the sender (`updateBackPressureRatio` in `replicationConnection.ts`) and read **only** by `clusterStatus.ts` — it never reaches the orchestrator.
- One `worker.on('exit')` is attached **per (db,node) subscription** on a shared worker object (`subscriptionManager.ts`, see the `worker?.on('exit', ...)` in the subscription path) → `MaxListenersExceededWarning` past ~10 databases (#357). The same anti-pattern (logical entity registers a listener on a long-lived shared emitter, nobody removes it deterministically) drives `forceResubscribe` leaking into persistent `subscriptions-updated` listeners (#349).

## Design direction
1. **Move connection state into shared memory.** Add connection-state, last-error-code, and a heartbeat-timestamp slot to the per-(db,node) status buffer (slots 9–15 are currently reserved), so the main thread reads truth directly instead of inferring it. This is the same channel #218 needs for back-pressure.
2. **A liveness watchdog owns `connected`.** No bytes + no pong within the ping timeout ⇒ down; a successful ping ⇒ up — independent of whether `'close'` ever fired.
3. **One `'exit'` handler per worker**, consulting the registry, instead of one per subscription.
4. **Idempotent resubscribe**, with one-shot intents detached from connection lifetime (no binding to persistent listeners).
5. **Reset connection state on restart** so outbound connections re-derive truth at boot.
6. **`remove_node` synchronously closes sockets** and drops registry entries; `cluster_status` reports real membership (replace the hardcoded `is_enabled:true`).

## Scope
- [ ] Extend shared-memory status schema: connection-state, last-error, heartbeat slots
- [ ] Watchdog-owned `connected` state machine (liveness-derived)
- [ ] Per-worker exit/reassign handler (replace per-subscription listeners)
- [ ] Idempotent resubscribe; detach `subscriptions-updated` from connection lifetime
- [ ] Reset connection state on restart
- [ ] `remove_node` → synchronous socket close + entry removal + real membership flag
- [ ] Demote the point-fix watchdogs (#424, #453-family, #463, #466) to telemetry/assertions once level-triggered truth owns recovery

## Retires / advances
- [ ] #289 — outgoing peers show connected:false after restart despite reachable
- [x] #233 / #420 — outbound WS dies without `'close'` *(closed via edge-triggered watchdogs #424/#466 — this workstream replaces those layers with level-triggered truth and retires the bug class)*
- [ ] #357 — worker.on('exit') per subscription → MaxListenersExceededWarning
- [ ] #349 — forceResubscribe leaks into persistent listeners (lifecycle half)
- [ ] #217 — removed node should indicate removal + close sockets
- **Enables:** #218 (W5), W6 (dedicated threads), W13 watchdog consolidation (#510)

## Dependencies
None upstream. Strongly recommended *before* W5/W6.

## Effort / risk
**L / medium.** Highest leverage in the epic.

## Acceptance criteria
- `connected` reflects reachability within the ping timeout regardless of whether `'close'` fired.
- A cluster with >10 databases produces no MaxListeners warning.
- A removed node reports removed and its sockets are closed without a restart.
- The main thread can read connection state and back-pressure from shared memory.
- The layered point-fix watchdogs are no longer load-bearing recovery paths — they fire as telemetry/assertions only.

---
🤖 Filed by Claude on behalf of Kris.