HarperFast / HarperFast/harper-pro
Filtered base copy for out-of-retention add_node start_time
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
`add_node { start_time }` limits a user database to data written after the cutoff. This works while the cutoff is inside the audit-retention window (the incremental audit-replay path honors it). When the cutoff predates retention, `shouldForceBaseCopyForRetention` upgrades the request to a base copy, and that base copy is **unfiltered**, so the whole backlog is re-sent. The "only data after this time" contract is then violated for any cutoff older than retention.
This is the same defect codex/@kriszyp raised on the PR that fixed the snake/camel `start_time` bug ([replicationConnection.ts:4463-4474](../blob/main/replication/replicationConnection.ts) rewrites the requested start to `0`, and the base-copy loop then sends every row). It is filed as a separate issue because a naive filter is unsafe: an initial attempt (carry an explicit cutoff on the subscription; skip base-copy rows where `(localTime ?? version) <= cutoff`) was pulled after an adversarial review found three data-loss paths.
## Why a naive filter loses data
1. **Self-catchup / resume cursors, not cutoffs.** `node.startTime` (camelCase) is overloaded: config routes use it as an operator cutoff ([knownNodes.ts:1170](../blob/main/replication/knownNodes.ts)), but [subscriptionManager.ts:782-788](../blob/main/replication/subscriptionManager.ts) pushes a self-catchup subscription whose `startTime` is a **durable resume cursor** (only when `REPLICATION_FAILOVER` is on). Sourcing the base-copy cutoff from `startTime` folds that resume cursor into the filter, so a failover node doing a base copy from a peer with no `start_time` would skip every row older than its resume cursor. Silent data loss.
2. **Merged multi-source walk.** One connection can carry subscriptions for several source nodes with different (or no) cutoffs. A single `min` cutoff applied to the merged base-copy walk truncates rows a no-cutoff co-subscription asked for.
3. **System database.** `iterateReplicatedDatabases` includes `system`, and the cutoff would strip `hdb_user` / `hdb_role` / `hdb_nodes` / schema rows written before the cutoff, so the joining node comes up without cluster metadata. (The shipped fix already guards the system database in the incremental path; a base-copy filter must do the same.)
## Constraints for a correct implementation
- Source the cutoff **only** from a genuine operator `start_time` (add_node snake `start_time` or a config-route cutoff for a real peer), never from a self-catchup / resume cursor. The self-catchup subscription is identifiable by `name === thisNodeName`.
- **Never** filter the `system` database (always full-copy cluster metadata).
- Scope the cutoff **per source node**, not one `min` over the merged walk, so a co-subscription that wants a full copy is not truncated. This is the hard part: the base copy is a single key-ordered walk of the database and does not currently carry per-source state.
- Preserve the post-copy audit replay from `copyStartTime` so writes concurrent with the copy are still delivered.
- Regression coverage must include: cutoff older than retention (the base-copy path), a `REPLICATION_FAILOVER` self-catchup concurrent with a base copy, a mesh connection with mixed/no cutoffs, and the system database.
## Current behavior (shipped)
Until this lands, a cutoff older than retention falls back to a full copy (the pre-existing, safe behavior). Operators who need to bound the initial volume should keep the cutoff inside the audit-retention window.
Lavinia, via Claude
Contributor guide
Research direction
Read replication/replicationConnection.ts around the base-copy loop and shouldForceBaseCopyForRetention, then trace subscription state through knownNodes.ts and subscriptionManager.ts. Define regression coverage for an older-than-retention cutoff, failover self-catchup, mixed mesh cutoffs, and the system database; done means filtered user data, full system metadata, and preserved post-copy audit replay.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100