HarperFast / HarperFast/harper
Replication bounded base-copy resync silently destroys receiver-only rows (no merge, no guard, no txns) — confirmed data loss on a 2-node 5.2.0 cluster
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
### Summary
On a 2-node 5.2.0 cluster, Harper's `forcing a bounded base-copy resync` path (triggered when a peer requests replication from a point older than retained transaction-log history) **silently destroyed rows that existed only on the receiving side** — a table-level replacement with no merge, no row-count guard, no data-loss warning, and no per-row transactions (the receiving node's transaction log shows only a bare `reload` entry, matching the peer's base-copy log line to within 2 seconds).
### Incident shape (fully forensically reconstructed)
- Inter-node replication for one database flapped repeatedly (ws :9933 disconnect storms, code 1006 / ECONNRESET) and stayed down past the 72h (`259200000ms`) txn retention — so every reconnect logged `requested replication ... which predates retained transaction-log history ... forcing a bounded base-copy resync`. Four such base-copies in three weeks on this pair.
- Rows written to node A during a down period (never replicated to B) were erased when a subsequent base-copy replaced the table. Ten rows lost; recovered only because `read_transaction_log` still held their upsert payloads.
- Aggravating: an external source re-syncing a SUBSET of rows (a federation spoke) masked the blast radius by healing its own rows within minutes — making the loss look scoped to one writer rather than 'everything after the divergence point'.
### Asks
1. Base-copy resync must not silently discard receiver-only rows: either merge (preserve rows newer than the divergence point), or refuse and demand an explicit operator flag, or at minimum snapshot the receiver's table first and log a LOUD row-count delta.
2. Row-count / recency guard: if a resync would reduce row count or delete rows newer than the base, alarm before applying.
3. Surface chronic re-base-copying as cluster ill-health (N base-copies of the same db within retention window = replication is effectively down); today it's a warn-level line nobody sees while fleet health elsewhere reports green.
4. Retention interplay: when replication downtime exceeds txn retention, the system currently guarantees data loss on the less-lucky node. Consider retention-aware backpressure or at least a documented operational bound.
Log-line shapes, timestamps, and the reload-txn correlation available on request (2-node Fabric-hosted cluster, harper-pro 5.2.0).
Contributor guide
Research direction
Start by tracing the bounded base-copy resync path described in the issue, including the retained transaction-log history check and the receiver's bare reload entry. Use read_transaction_log and the reported row-count and recency behavior to establish a reproducible case. Done should include protection against silent receiver-only row loss, an explicit warning or refusal path, and visibility into repeated base-copy resyncs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100