HarperFast / HarperFast/harper-pro

Filtered base copy for out-of-retention add_node start_time

Open
#727 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

`add_node { start_time }` limits a user database to data written after the cutoff. This works while the cutoff is inside the audit-retention window (the incremental audit-replay path honors it). When the cutoff predates retention, `shouldForceBaseCopyForRetention` upgrades the request to a base copy, and that base copy is **unfiltered**, so the whole backlog is re-sent. The "only data after this time" contract is then violated for any cutoff older than retention.

This is the same defect codex/@kriszyp raised on the PR that fixed the snake/camel `start_time` bug ([replicationConnection.ts:4463-4474](../blob/main/replication/replicationConnection.ts) rewrites the requested start to `0`, and the base-copy loop then sends every row). It is filed as a separate issue because a naive filter is unsafe: an initial attempt (carry an explicit cutoff on the subscription; skip base-copy rows where `(localTime ?? version) <= cutoff`) was pulled after an adversarial review found three data-loss paths.

## Why a naive filter loses data

1. **Self-catchup / resume cursors, not cutoffs.** `node.startTime` (camelCase) is overloaded: config routes use it as an operator cutoff ([knownNodes.ts:1170](../blob/main/replication/knownNodes.ts)), but [subscriptionManager.ts:782-788](../blob/main/replication/subscriptionManager.ts) pushes a self-catchup subscription whose `startTime` is a **durable resume cursor** (only when `REPLICATION_FAILOVER` is on). Sourcing the base-copy cutoff from `startTime` folds that resume cursor into the filter, so a failover node doing a base copy from a peer with no `start_time` would skip every row older than its resume cursor. Silent data loss.

2. **Merged multi-source walk.** One connection can carry subscriptions for several source nodes with different (or no) cutoffs. A single `min` cutoff applied to the merged base-copy walk truncates rows a no-cutoff co-subscription asked for.

3. **System database.** `iterateReplicatedDatabases` includes `system`, and the cutoff would strip `hdb_user` / `hdb_role` / `hdb_nodes` / schema rows written before the cutoff, so the joining node comes up without cluster metadata. (The shipped fix already guards the system database in the incremental path; a base-copy filter must do the same.)

## Constraints for a correct implementation

- Source the cutoff **only** from a genuine operator `start_time` (add_node snake `start_time` or a config-route cutoff for a real peer), never from a self-catchup / resume cursor. The self-catchup subscription is identifiable by `name === thisNodeName`.
- **Never** filter the `system` database (always full-copy cluster metadata).
- Scope the cutoff **per source node**, not one `min` over the merged walk, so a co-subscription that wants a full copy is not truncated. This is the hard part: the base copy is a single key-ordered walk of the database and does not currently carry per-source state.
- Preserve the post-copy audit replay from `copyStartTime` so writes concurrent with the copy are still delivered.
- Regression coverage must include: cutoff older than retention (the base-copy path), a `REPLICATION_FAILOVER` self-catchup concurrent with a base copy, a mesh connection with mixed/no cutoffs, and the system database.

## Current behavior (shipped)

Until this lands, a cutoff older than retention falls back to a full copy (the pre-existing, safe behavior). Operators who need to bound the initial volume should keep the cutoff inside the audit-retention window.

Lavinia, via Claude

Contributor guide

Open the contributing guide

Research direction

Read replication/replicationConnection.ts around the base-copy loop and shouldForceBaseCopyForRetention, then trace subscription state through knownNodes.ts and subscriptionManager.ts. Define regression coverage for an older-than-retention cutoff, failover self-catchup, mixed mesh cutoffs, and the system database; done means filtered user data, full system metadata, and preserved post-copy audit replay.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.