HarperFast / HarperFast/harper-pro
Nothing verifies the range a copy-resume cursor claims was already delivered
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
A bulk-copy resume trusts the follower's persisted cursor: everything in copy order before `{currentTable, afterKey}` is assumed delivered, and the sender skips straight past it. Nothing ever verifies that assumption against actual table content. When a cursor outruns delivery (the harper-pro#537 poisoned-cursor class, or any future variant), every resume re-seals the hole: the skipped range is never re-examined, the counts sit thousands of rows apart, and no signal fires anywhere.
The reworked stream-count verification (harper-pro#538) covers in-flight loss within one copy session. It cannot see at-rest holes from earlier sessions, which is exactly where poisoned cursors live. In the #538 review we converged on a deterministic per-range key checksum as the trustworthy at-rest check.
## Proposal
On resume (and only on resume; fresh copies claim nothing), the sender checksums the key range the cursor claims was delivered: every prior table in copy order, plus the resume table through `afterKey` inclusive. Keys only, primary-key order, local-only records excluded, capped, wall-clock paced. It sends the per-table checksums together with the exact bounds and cap it used, right after COPY_START, as a new `COPY_RANGE_CHECKSUM` message. The follower computes the same checksums over the identical range (the payload's bounds, not a re-read of its own cursor, which may have advanced or been removed since the request) and compares.
Alert-only: a mismatch logs an error with exact per-table key counts and stashes the details on the connection. No forced re-copy (against receiver-side undecodable drops a re-copy loops forever) and no cursor surgery. Known benign-drift sources under live traffic (rows inserted or deleted in the range since the copy began, live deliveries from other sources, local writes) make small drift expected on busy tables; a poisoned cursor shows as a massive one-sided gap.
Old senders and receivers interoperate unchanged: the message id is new and the command switch ignores unknown ids.
## Cross-references
- harper-pro#537 observed divergence; harper-pro#538 review thread (checksum agreed as the follow-up).
*Lavinia, via Claude*
Contributor guide
Research direction
Start by tracing the existing COPY_START handling and the command switch that processes message ids. Read how copy-resume cursors and stream-count verification are represented, then define the checksum message's bounds and comparison flow; done means resume ranges are checked, mismatches are logged with counts and connection details, and unknown message ids still interoperate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100