HarperFast / HarperFast/harper

Replication resume skips an origin's post-restart writes when its transaction-log clock moved backwards

Open
#2,629 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

A RocksDB transaction log's keys are claimed from a process-wide monotonic clock that is **not persisted across restarts**: rocksdb-js `docs/transaction-log.md` states that "a restart after the wall clock moved backwards can reissue" a key. Replication resume filters the origin's log by key (`replication/replicationConnection.ts` sender: `getRange({ start: cursor, exclusiveStart: true, ... })`), so after such a restart every entry the origin appends with a key **below a subscriber's persisted cursor** is never delivered to that subscriber on reconnect. The live tail (no reconnect) is unaffected: a per-log range read yields entries in file/append order, filtered by key — verified empirically on rocksdb-js 2.9.0 with out-of-order timestamps `100, 200, 101, 300`: `start>50` yields `100, 200, 101, 300`; `start>150` yields `200, 300` (the post-rollback `101` is skipped).

Consequence: a peer that reconnects after an origin's clock rollback silently misses that origin's post-rollback writes until the origin's clock passes the peer's cursor again. Nothing logs it; the peer's status shows a healthy, advancing cursor.

## Proposed fix (rocksdb-js, small)

Persist a **clock floor**: on open, the monotonic timestamp source starts at `max(now, highest key in the local log + 1 tick)` (the running-maxima index the reader already builds has that value). Keys then never regress across restarts, replay-by-key equals append order, and cursors stay sound. The alternative — an incarnation-qualified append sequence carried on the wire — fixes the same thing at the cost of a wire/storage migration and is not proposed.

## Why it matters now

harper-pro#822 (cluster record locks) proves successor freshness with a `lockBarrier` entry (harper#2625) the origin appends after the fenced write; that proof holds on the live path and fails closed on resume (the barrier is simply never delivered → 503), so locks are not made unsafe by this — but ordinary replication is silently incomplete in the same window, which is the larger problem.

## Not in scope

harper-pro's lock transport (harper-pro#822). Deduplication of reissued keys (the existing `(timestamp, origin)` identity rule stands).

Contributor guide

Open the contributing guide

Research direction

Start with docs/transaction-log.md and replication/replicationConnection.ts, especially the sender's getRange call and the local log's running-maxima index. Trace how the timestamp source is initialized on open and how the persisted cursor is used during resume. Done means keys do not regress across restarts and a reconnect delivers post-restart writes after a clock rollback.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.