HarperFast / HarperFast/harper
Replication resume skips an origin's post-restart writes when its transaction-log clock moved backwards
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
A RocksDB transaction log's keys are claimed from a process-wide monotonic clock that is **not persisted across restarts**: rocksdb-js `docs/transaction-log.md` states that "a restart after the wall clock moved backwards can reissue" a key. Replication resume filters the origin's log by key (`replication/replicationConnection.ts` sender: `getRange({ start: cursor, exclusiveStart: true, ... })`), so after such a restart every entry the origin appends with a key **below a subscriber's persisted cursor** is never delivered to that subscriber on reconnect. The live tail (no reconnect) is unaffected: a per-log range read yields entries in file/append order, filtered by key — verified empirically on rocksdb-js 2.9.0 with out-of-order timestamps `100, 200, 101, 300`: `start>50` yields `100, 200, 101, 300`; `start>150` yields `200, 300` (the post-rollback `101` is skipped).
Consequence: a peer that reconnects after an origin's clock rollback silently misses that origin's post-rollback writes until the origin's clock passes the peer's cursor again. Nothing logs it; the peer's status shows a healthy, advancing cursor.
## Proposed fix (rocksdb-js, small)
Persist a **clock floor**: on open, the monotonic timestamp source starts at `max(now, highest key in the local log + 1 tick)` (the running-maxima index the reader already builds has that value). Keys then never regress across restarts, replay-by-key equals append order, and cursors stay sound. The alternative — an incarnation-qualified append sequence carried on the wire — fixes the same thing at the cost of a wire/storage migration and is not proposed.
## Why it matters now
harper-pro#822 (cluster record locks) proves successor freshness with a `lockBarrier` entry (harper#2625) the origin appends after the fenced write; that proof holds on the live path and fails closed on resume (the barrier is simply never delivered → 503), so locks are not made unsafe by this — but ordinary replication is silently incomplete in the same window, which is the larger problem.
## Not in scope
harper-pro's lock transport (harper-pro#822). Deduplication of reissued keys (the existing `(timestamp, origin)` identity rule stands).
Contributor guide
Research direction
Start with docs/transaction-log.md and replication/replicationConnection.ts, especially the sender's getRange call and the local log's running-maxima index. Trace how the timestamp source is initialized on open and how the persisted cursor is used during resume. Done means keys do not regress across restarts and a reconnect delivers post-restart writes after a clock rollback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100