HarperFast / HarperFast/harper

Replication bounded base-copy resync silently destroys receiver-only rows (no merge, no guard, no txns) — confirmed data loss on a 2-node 5.2.0 cluster

Open
#2,193 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

### Summary

On a 2-node 5.2.0 cluster, Harper's `forcing a bounded base-copy resync` path (triggered when a peer requests replication from a point older than retained transaction-log history) **silently destroyed rows that existed only on the receiving side** — a table-level replacement with no merge, no row-count guard, no data-loss warning, and no per-row transactions (the receiving node's transaction log shows only a bare `reload` entry, matching the peer's base-copy log line to within 2 seconds).

### Incident shape (fully forensically reconstructed)

- Inter-node replication for one database flapped repeatedly (ws :9933 disconnect storms, code 1006 / ECONNRESET) and stayed down past the 72h (`259200000ms`) txn retention — so every reconnect logged `requested replication ... which predates retained transaction-log history ... forcing a bounded base-copy resync`. Four such base-copies in three weeks on this pair.
- Rows written to node A during a down period (never replicated to B) were erased when a subsequent base-copy replaced the table. Ten rows lost; recovered only because `read_transaction_log` still held their upsert payloads.
- Aggravating: an external source re-syncing a SUBSET of rows (a federation spoke) masked the blast radius by healing its own rows within minutes — making the loss look scoped to one writer rather than 'everything after the divergence point'.

### Asks
1. Base-copy resync must not silently discard receiver-only rows: either merge (preserve rows newer than the divergence point), or refuse and demand an explicit operator flag, or at minimum snapshot the receiver's table first and log a LOUD row-count delta.
2. Row-count / recency guard: if a resync would reduce row count or delete rows newer than the base, alarm before applying.
3. Surface chronic re-base-copying as cluster ill-health (N base-copies of the same db within retention window = replication is effectively down); today it's a warn-level line nobody sees while fleet health elsewhere reports green.
4. Retention interplay: when replication downtime exceeds txn retention, the system currently guarantees data loss on the less-lucky node. Consider retention-aware backpressure or at least a documented operational bound.

Log-line shapes, timestamps, and the reload-txn correlation available on request (2-node Fabric-hosted cluster, harper-pro 5.2.0).

Contributor guide

Open the contributing guide

Research direction

Start by tracing the bounded base-copy resync path described in the issue, including the retained transaction-log history check and the receiver's bare reload entry. Use read_transaction_log and the reported row-count and recency behavior to establish a reproducible case. Done should include protection against silent receiver-only row loss, an explicit warning or refusal path, and visibility into repeated base-copy resyncs.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.