HarperFast / HarperFast/harper-pro

Forced base-copy resync cannot repair a missed DELETE: the copy emits puts only, so the follower keeps the stale row

Open
#794 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

The sender's bounded base-copy resync walks the primary store and emits **puts only**. When the entry a follower missed was a **delete**, the copy cannot repair it: the follower keeps the stale row and reports converged.

Two triggers force this copy, and both have the gap:

1. **Retention gap** — the follower's resume cursor predates the transaction-log history the sender still retains (the `shouldForceBaseCopyForRetention` path, `replication/replicationConnection.ts`). A delete purged from the log is never re-sent.
2. **Mid-log transaction-log break** — the follower is stranded behind a corrupt frame and the sender forces the copy (the `CorruptFrameResync` path, added for HarperFast/harper#2016 / HarperFast/harper#2063). If the torn frame held a delete, the copy re-sends every surviving row but nothing removes the deleted one on the follower.

## Why it matters

The invariant these two paths enforce today is "every acknowledged **put** reaches the follower". A delete that fell in the missed span leaves a row on the follower that the source no longer has, indefinitely, with no signal: `cluster_status` reports the follower caught up and the copy completes cleanly. It was accepted as a documented limitation when the break-forced copy landed (the same lever the retention-forced copy already had), with this issue as the follow-up.

## Where to look

- The copy walk: `replicateOverWS` → the `currentSequenceId === 0` branch → `COPY_START` … `COPY_COMPLETE`. Records are emitted from `primaryStore` in key order; there is no tombstone or key-manifest phase.
- The break trigger: `CorruptFrameResync.afterDrain()` → `currentSequenceId = 0`.
- The retention trigger: `shouldForceBaseCopyForRetention`.

## Options

- **Key manifest**: during the copy, stream the set of primary keys (or per-table key ranges) the sender holds; on `COPY_COMPLETE` the follower deletes rows it holds for that subscription that are absent from the manifest. Converges deletes for both triggers; costs one extra pass over keys.
- **Tombstone sweep**: the sender emits a delete for each record whose latest audit entry is a delete within the window the copy is standing in for. Cheaper, but the retention trigger has no window to sweep (the entries are gone), so it only covers the break trigger.
- **Document only**: keep the puts-only copy and state the limitation. This is where things stand today.

## Acceptance

- A delete that lands in the missed span (torn frame, or purged by retention) is applied on the follower after the forced copy, for both triggers.
- `integrationTests/cluster/txnlogTearReplication.test.mjs` gains a torn-DELETE variant (tear the frame that holds a delete; assert the row is gone on B), and the retention-gap test gains the purged-delete equivalent.
- The existing puts-only guarantees stay green.

Contributor guide

Open the contributing guide

Research direction

Read replicateOverWS around the currentSequenceId === 0 branch, then trace CorruptFrameResync.afterDrain() and shouldForceBaseCopyForRetention. Run integrationTests/cluster/txnlogTearReplication.test.mjs and locate the retention-gap test. Done means forced copies remove rows deleted in the missed span for both triggers while existing put guarantees remain green.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs, typescript
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.