Replication: subscriber silently skipped ~700k backlogged writes during a write-conflict storm — cursor stayed "current", no base-copy triggered, no error anywhere

## Summary

During a bulk table migration on a 4-node cluster, the replication stream for one database **silently dropped ~700k of 803k writes** on all three subscribing peers. Every health surface looked fine: sockets connected, backpressure 0, `lastReceivedRemoteTime` current, and **new writes kept replicating normally** after the incident window. No base-copy/resync was triggered and no error or warning was logged on any node. The gap was only discovered by manually comparing `SELECT COUNT(*)` across nodes.

This is data-loss-grade divergence with a fully healthy-looking cluster — a concrete field case for #432 (cursor correctness & divergence detection).

## Environment

- harper-pro **5.1.26** (docker `harperfast/harper-pro:5.1.26`)
- 4-node Fabric cluster `prerender.kohls.harperfabric.com` (yc0-us-central-1 / cd5-us-southeast-1 / e9v-us-east-1 / v3t-us-sea-1)
- Database `render_service`, table `Target` (string PK `url`), replicated (no residency pinning on this table)

## Timeline (2026-08-02, ~17:00–19:30Z)

1. A component migration on **yc0** bulk-wrote **803,324** rows into `render_service.Target` (chunked writes over ~30 min).
2. Concurrently, duplicate sweeps (same component, other workers) raced the migration and died with repeated **"After 40 retries, unable to commit transaction"** — i.e., the table was under a sustained write-conflict storm while the subscriber stream was consuming the same backlog.
3. Peers ended up with **103,658** rows each. Delivery stopped **mid-backlog**, roughly coincident with the conflict storm.
4. Writes issued *after* the window replicated instantly and correctly (verified with a control-row write that appeared on all peers within seconds).

## Key observations

- **The skipped range is not a clean prefix.** Peer copies had holes: rows at higher PK offsets were present while earlier rows were missing — consistent with the subscriber advancing its resume cursor past transactions it never applied (or the sender skipping log segments), rather than a simple disconnect.
- `cluster_status` on all nodes: `connected: true`, backpressure 0, `lastReceivedRemoteTime` current — before, during, and after.
- **No** "predates retained transaction-log history… forcing a bounded base-copy resync" lines anywhere (leader-side stale-cursor handler, `replication/replicationConnection.js` ~line 3046 in the 5.1.26 dist, never fired). No full-copy triggered either (the no-resume-cursor path near ~4160, where the #426 comment lives, correctly did not apply — cursors existed and looked "valid").
- So the cluster considered the cursors valid while ~700k committed, audited transactions between them and head were never delivered/applied — and nothing detected the divergence.

## Impact

3 of 4 nodes served a 103k-row view of an 803k-row table for ~2.5 h. Downstream logic that treats a local miss as "row does not exist" took destructive actions (deleting sibling schedule rows). Nothing in Harper surfaced the problem; discovery was a human noticing a wrong dashboard count.

## Suspected area / repro hypothesis

Bulk sequential writes to one table + concurrent conflicting writers on the same keys (enough to produce "After 40 retries, unable to commit transaction" aborts) while ≥1 subscriber is consuming the resulting backlog over the per-database stream. Suspect the interplay between the conflict/retry machinery's audit entries and the subscriber's cursor advance — the cursor appears to have jumped over a large undelivered span while remaining "valid" to the leader's retention check.

## Recovery that worked (for the record)

Re-emitting every row from the holder via ops-API `upsert` with the **complete record** (161 × 5k chunks) pushed the backlog through the same stream successfully; peers converged to 803,324 with zero retries. Note for anyone repeating this: `update`/`upsert` on an existing row audit as a **patch delta of only the supplied fields** (`core/resources/Table.js` ~1887), and a subscriber missing the base row stores the patch body verbatim (`tracked.js updateAndFreeze`: `if (!target) return changes`) — so a partial-field "touch" re-emit would have materialized field-stripped stubs on the peers. Re-emits must carry every field.

## Asks

1. Root-cause the cursor advance across an undelivered span under write-conflict storms.
2. Divergence detection (#432): even a cheap periodic per-table count/hash comparison between publisher and subscriber would have caught this in minutes instead of hours.

---

*Filed by an agent (Claude) from the incident session; evidence gathered live from the cluster (UDS ops, docker logs, 5.1.26 dist source).*

Start by tracing cursor advancement and retention validation in replication/replicationConnection.js around lines 3046 and 4160, alongside the conflict/retry audit path described in the report. Use the bulk-write and conflicting-writer hypothesis to reproduce the skipped span, then inspect core/resources/Table.js around line 1887 and tracked.js updateAndFreeze for recovery behavior. Done means the cursor-loss cause is established and divergence detection or a regression test addresses the failure.

Written by the indexing model from the issue text.