Mesh replication amplifies a re-delivered `hdb_nodes` delete into millions of duplicate audit entries; the resulting same-version run permanently exceeds `replication_maxPayload`

**Repo:** harper-pro. **Observed on:** harper-pro 5.2.7, RocksDB storage, 16-node full-mesh production cluster (wm-usgm-er-prod). A prior occurrence on the same cluster on 2026-08-17 (~394 MB frames) self-healed when the entries aged out of `auditRetention` (3d). Related to but distinct from #711.

## Summary

Two defects compound:

**Defect A (amplification): a replicated `delete` of a record that does not exist locally is re-applied and re-logged on every delivery, with no dedup.** Every dedup layer requires a stored record to tie against: `isDurableIdentityTie` returns false when `!existing` (replicationConnection.ts ~1113), the leading-duplicate fast-skip calls it through `readLocalEntry` (~3780), and the CRDT resequencing walk with its keyed `auditStore.get(txnTime, tableId, id, nodeId)` dedup (Table.ts ~2602, ~2667) only runs when an `existingEntry` is present. A delete of an absent row hits none of these, applies as a no-op, and appends a fresh audit entry keyed at `localTime = version` (the apply transaction inherits the event's timestamp: Table.ts ~944, transaction.ts ~52). In a mesh, each appended copy is new log tail that every other leg picks up and forwards; each receiver re-logs it again. The result is exponential echo: one delete entry became ~6.4 million byte-identical audit entries per node in 16 minutes.

**Defect B (no flush inside a same-version run): the audit send path cannot ship a run of entries sharing one `version`, no matter its size.** `sendAuditRecord` flushes only when `auditRecord.version` changes (~4972) or on the synthesized `end_txn` at iterator drain (~5508). #711 added `COPY_FLUSH_BYTES` pacing to the bulk-copy path; the audit-replay path has no equivalent, so a same-version run larger than `MAX_PAYLOAD` accumulates whole, `checkExcessMessageSize` throws at the next version change (~5060, the deliberate no-data-loss throw), the leg closes, resumes from the un-advanced cursor, and re-encodes the same run forever. Every duplicate shares one `localTime` key, so no cursor value can express progress inside the run — a size-based flush needs sub-key resume semantics, which is presumably why the flush-on-version-change design exists. Note this also wedges on any legitimately huge single transaction.

A third contributing behavior: the duplicates are appended at `localTime = version`, i.e. **behind** newer keys already in the log, violating the txnlog's monotonic-timestamp assumption ("fixed size index entries for fast traversal using binary search"). Resume cursors newer than the run's key do not skip it, because iteration past the binary-searched start position is append-ordered.

## Production evidence (wm-usgm-er-prod, 2026-09-01)

- Seed: the bridge peer row `hdb_nodes/wm-usgm-stage-us-sea01.harperdbcloud.com` is caught in a repeating authenticated `remove_node`/re-add cycle on the bridge node (tracked separately). One such delete committed at version `1788239374542.4155` (05:09:34.542Z), present exactly once in the origin node's local log.
- 06:08:00Z: mesh-wide reconnect wave (all 15 peers × data/system/redirects legs re-established, some twice).
- 06:12:19–06:28:15Z: every non-bridge node's `eva-us-sea-1` (origin) transaction log grew from ~16 MB to 725–800 MB — e.g. us-west-2: 50 × 16 MiB `.txnlog` files. Parsing them offline: ~6.4M entries that are **byte-identical** `delete` entries for that row, all with version `1788239374542.4155`, all keyed at that same timestamp, `user_name` = the bridge's replication credential. The origin node holds the delete once and has zero duplicates (74 MB local log) — receivers minted every copy.
- 06:22:14Z onward: `Message too large to send, size: ... database: system` from 15 of 16 nodes; sizes grow 470,492,640 → 678,678,033 while echo inflow continues, then freeze (us-west-2: 678,695,309 ≈ 6.4M entries × ~105 wire bytes). 39–481 errors/hour/node since (warn-throttled).
- Legs pinned: `lastReceivedRemoteTime` shows 05:38:31Z on 14 of 15 peers per node (the max-version watermark from the last successfully applied writes — two `hdb_nodes` patches). Only the origin's direct legs still deliver (its log has no run).
- The 100 MB cap is what stopped the echo: once a node's run exceeded `MAX_PAYLOAD` its fan-out of the run wedged, and by ~06:34 all fan-out legs were wedged. The wedge is the storm's only circuit breaker.
- Precedent: identical error signature on 2026-08-17 (~394 MB), gone after retention purged the entries. The current run ages out of the 3-day window on ~2026-09-04.

## Reproduction sketch

1. 3+ node v5.2.7 mesh (RocksDB), any replicated database.
2. Write and replicate a row; then delete it, so a `delete` audit entry (version V) exists in every node's origin log for node A within the retained window.
3. Force reconnect churn such that the delete is re-delivered outside the leading-duplicate tie window (e.g. bounce legs, or re-deliver via an indirect leg whose per-origin startTime predates V — indirect legs rarely advance per-origin cursors for origins they don't primarily serve).
4. Observe: each receiver re-applies the no-op delete and appends a fresh audit entry at key V; peers pick each other's copies up and re-log again; copies multiply per exchange round.
5. Once a node's same-version run exceeds `replication_maxPayload`, every leg carrying it throws `Message too large to send` at `sendQueuedData` (called from the version-change flush in `sendAuditRecord`), closes, resumes from the un-advanced cursor, and repeats indefinitely. The database's mesh replication is permanently wedged while inflow from not-yet-wedged legs keeps growing the run.

## Impact

- Full system-db (or any-db) mesh replication outage: node/user/role/schema state stops propagating cluster-wide.
- The retry loop re-encodes hundreds of MB per attempt per leg ("JavaScript execution has taken too long" watchdogs fire).
- Self-heals only by `auditRetention` file purge, days later, and only until the next delete is re-delivered.
- Raising `replication_maxPayload` un-chokes the echo and makes it worse (see below), so there is no safe config workaround.

## Suggested fixes

1. **Dedup replicated deletes (primary).** On applying a replicated delete with no stored record, consult the audit log for an existing entry at `(version, tableId, id, nodeId)` — the same keyed lookup the CRDT path already uses at Table.ts ~2667 — before appending a new audit entry; skip apply+relog on a hit. Equivalently: extend `isDurableIdentityTie`/the leading-dup window with a per-origin version floor so `version <= floor` deletes of absent rows are skippable. Deletes are currently the only entry class with zero dedup layers.
2. **Bound the audit-path frame (the #711 analog).** A size-based flush inside a same-version run requires resumable sub-positions (e.g. counting entries within a `localTime` key in the sequence update, or splitting the run into multiple frames the receiver treats as one transaction). Absent that, at minimum detect the condition (frame > cap with unchanged txnTime) and surface a distinct, actionable error naming the run's version/table/id instead of retrying the identical encode forever.
3. **Reject out-of-order relog keys.** Appending at `localTime = version` behind newer keys breaks the log's binary-search contract and pins resume cursors; relogged entries could be keyed at arrival time (preserving `version` inside the entry) or the duplicate append avoided entirely per fix 1.

Fix 1 alone prevents the storm; fix 2 alone prevents the permanent wedge; both are worth having.

---
Lavinia, via Claude

Start with replicationConnection.ts around isDurableIdentityTie and readLocalEntry, then trace the CRDT dedup lookups in Table.ts and transaction timestamp handling in transaction.ts. Read sendAuditRecord, sendQueuedData, and checkExcessMessageSize to understand the same-version replay path; use the reproduction sketch to verify that absent-row deletes are not repeatedly re-logged and oversized runs do not retry forever.

Written by the indexing model from the issue text.