HarperFast / HarperFast/harper-pro
Mid-copy kill can resume via the direct seq cursor instead of the copy cursor, permanently skipping uncopied rows
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
A clone killed mid-bulk-copy can reconnect via the **direct seq-cursor path instead of resuming the copy**, permanently skipping the not-yet-copied rows. Observed live in CI ([Integration 3/3 on PR #657](https://github.com/HarperFast/harper-pro/actions/runs/31020139809/job/92359773619), `cloneResume.test.mjs`); rare — the same test passes locally and usually in CI.
## Evidence (from the failing run's log dump)
First clone attempt (killed at 15:45:25):
- `system`'s copy completed 15:44:28 (persisted shared-status watermark `1785944668706.3`).
- `data`'s copy was still in flight: its received-version watermark was **0 at the kill** (shared status survives restart on RocksDB — the second process still shows first-attempt-era stamps for `system`).
Restart at 15:45:26:
```
Connected to ws://127.0.0.3:9933, db: data
469-1c:9933 1 armed leading-duplicate fast-skip 127.0.0.3 1 data direct resume cursor 1785944669174
```
No `Requesting full copy of database data` — compare a healthy resume (local run of the same test), where the restart logs `Requesting full copy of database data` and completes. `1785944669174` ≈ 15:44:29.17, the copy-start era — a seq cursor persisted **mid-copy**.
Result: the leader replays nothing (the 4000 rows' audit versions predate the cursor), `data` receives zero frames for the whole 120s window, and the clone is missing most of the table with no pending work anywhere.
## Invariant that should hold
While a bulk copy is incomplete, the persisted resume state must never select the direct-cursor path on reconnect: either a copy cursor exists (copy resume) or no cursor exists (fresh full copy). A seq cursor persisted mid-copy (e.g. via `seqUpdateEndTxn` from a `SEQUENCE_ID_UPDATE` / empty-txn sequence update — those paths persist `[seq]` during the copy) combined with an absent/unusable copy cursor (e.g. `copyFromNodeId` undefined at `COPY_START`, or a kill before the first `COPY_CURSOR_FLUSH_INTERVAL_MS` flush) breaks it.
## Interaction with #655 / PR #657
Before the sync-monitor fix, this scenario either failed the test's row-count assert or, in production, silently produced a clone missing data while `Available`. With the monitor fix the clone correctly stays `Unavailable` (nothing ever arrives, the stall detector eventually fails the clone) — fail-safe, but the underlying skipped-copy state is the real defect. This is the live confirmation of the mid-copy cursor-advance hazard in the `SEQUENCE_ID_UPDATE` handler (same family as #426/#421).
## Repro direction
Needs a targeted repro forcing the window: kill the clone after the first mid-copy sequence-update persists `[seq]` but before the first copy-cursor flush (`REPLICATION_COPYCURSORFLUSHINTERVALMS` high + kill early), then restart and observe the direct-cursor reconnect with no copy re-request.
Contributor guide
Research direction
Start with cloneResume.test.mjs and the SEQUENCE_ID_UPDATE handler, especially seqUpdateEndTxn, COPY_START, and the copy-cursor flush path. Reproduce the kill window by delaying REPLICATION_COPYCURSORFLUSHINTERVALMS, then restart the clone and inspect whether it requests a full-copy or incorrectly takes the direct seq-cursor path. Done means an incomplete copy cannot reconnect through a direct cursor and the targeted test preserves all rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100