HarperFast / HarperFast/harper-pro

Replication: transitive/proxied re-delivery floods peers with already-applied out-of-order writes (reduce volume; complements harper#1310)

Open
#399 1 comment 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

Transitive/proxied replication re-delivers **already-applied** writes to peers **out of order**, generating a high volume of redundant out-of-order applies. On records with deep audit histories this drove the core out-of-order resequencing walk to its depth cap repeatedly, starving the event loop (→ replication ping-timeouts, wedged subscriptions). Observed in production on `[customer-cluster]`.

This is the **"reduce the volume"** counterpart to harper#1310 (which *absorbs* these cheaply in core via an up-front keyed dedup). #1310 stops them from being catastrophic, but they're still generated and still cost work every cycle — this issue tracks stopping them at the source.

## Evidence ([customer-cluster], [node] on 5.1.1)

A non-pausing logpoint at the core depth-cap site captured the capping writes:
- `type=patch`, `fullUpdate=false`, **every one carrying `viaNodeId`** (relayed via a proxy node, not direct).
- Multiple source nodes (`srcNode` 4/8/9), all **older than the local record head** (out-of-order).
- The post-cap keyed lookup matched (`dupVerMatch=true`, `dupNode===srcNode`) — i.e. they were **exact `(version, nodeId)` duplicates already applied**, redundantly re-delivered via the relay path, each forcing an ~858–1000-step walk before being discarded.

So the re-deliveries are real, redundant, and proxied.

## Suspected mechanism

The proxy-resume start-time derivation in `replication/replicationConnection.ts` (~the indirect-connection block that reads a proxy node's seqId, around the `Using sequence id from proxy node` path) appears to re-stream a longer already-applied tail than necessary — when the proxied seq cursor isn't relayed/persisted as tightly as the direct cursor, the leader streams from too-early a point and re-delivers writes the follower already has. (High confidence on the *symptom* — proxied out-of-order duplicate re-delivery — from the logpoint; the exact start-time-relay gap as the generator is the leading hypothesis from reading the resume code, not yet confirmed end-to-end.)

## Relationship to existing work

- **harper#1310** — up-front keyed dedup in `Table.commit` *absorbs* these in O(1) instead of an O(depth) walk. Mitigates impact; does not reduce volume.
- **#370** (leading-duplicate fast-skip) is **deliberately un-armed for proxied/indirect subscriptions** (`hasPersistedResumeCursor` checks the direct cursor only). Extending its arming to proxied subscriptions would give an earlier, cheaper skip for the *in-order* proxied subset.
- **#362 / harper#1275** — the shared-structure decode root cause (separate; already fixed).

## Proposed directions

1. **Tighten the proxy seq-cursor relay** so a proxied resume starts from an accurate point and doesn't re-stream an already-applied tail (attacks the cause).
2. **Arm #370's leading-dup-skip for proxied subscriptions** (per-origin-node cursor from the proxy seqId) — earlier skip for the in-order subset.
3. Consider whether redundant transitive delivery is needed at all when a follower already receives the writes directly (reduce proxied fan-in).

Filed from the [customer-cluster] incident investigation. Investigation + writeup by Claude (Opus 4.8).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.