HarperFast / HarperFast/harper-pro

Replication: tables created after a subscription are silently never replicated when replicateByDefault is false, with the resume cursor advancing over their records

Open
#677 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

When a peer subscribes with `replicateByDefault: false`, the table list it sends is an **inclusion** list snapshotted at subscription time. A table created *after* that subscription is established is therefore not in the list, and every audit record for it is silently skipped on the send side — **with the peer's durable resume cursor advancing over those records the whole time**.

The subscription is not re-posted when a table is added, so this does not self-heal. It persists for the life of the connection and is invisible: no error, no warning, `cluster_status` healthy, and the cursor stays "valid" so no base-copy resync is triggered.

Line references are against `main` @ `2aed45a9` (v5.2.1).

## Mechanism

**1. The list is a snapshot, and it is an inclusion list when `replicateByDefault` is false.**

The subscription request enumerates the local table registry once (`replication/replicationConnection.ts:5225-5253`). When a node has explicit `subscriptions` configured, `replicateByDefault` is forced off:

```js
replicateByDefault = false; // now turn off the default replication because it was an explicit list of subscriptions
```
— `replication/replicationConnection.ts:5246`

`replicateByDefault: false` also arises via `forReplicatedDatabase` for a database that is not in `options.databases` but contains explicitly-replicated tables (`replication/replicator.ts:797`).

**2. The send side gates every record on that snapshot.**

```js
const tableToTableEntry = (table) => {
if (
table &&
(firstNode.replicateByDefault
? !firstNode.tables.includes(table.tableName)
: firstNode.tables.includes(table.tableName))
) {
return { table };
}
};
```
— `replication/replicationConnection.ts:3538`

`firstNode` is `nodeSubscriptions[0]`, captured when the subscription is handled and never refreshed. With `replicateByDefault: true` the list is an *exclusion* list, so new tables replicate by default and everything is fine. With it false, a table absent from the snapshot can never match.

**3. A non-match is a silent skip that advances the cursor.**

`tableToTableEntry` returns `undefined`, so the record falls into:

```js
logger.debug?.('Not subscribed to table', tableId);
return skipAuditRecord();
```
— `replication/replicationConnection.ts:3605-3606`

`skipAuditRecord()` (`:3723`) arms a 300 ms timer that sends `[SEQUENCE_ID_UPDATE, currentSequenceId]` (`:3732`). The receiver converts that into an `end_txn` (`:3112`) and core's apply loop persists it as the durable resume cursor (`core/resources/Table.ts:757`, applied at `:856`). The skipped records are now permanently behind the cursor.

**4. Adding a table does not re-post the subscription.**

`onUpdatedTable` does reach the subscription manager — `forEachReplicatedDatabase` registers `onUpdatedTable((Table) => forReplicatedDatabase(Table.databaseName, ...))` (`replication/replicator.ts:775`) — which calls `onDatabase(databaseName, tablesReplicateByDefault, forceResubscribe = false)`.

But for an already-subscribed database that path returns early:

```js
if (existingEntry) {
worker = existingEntry.worker;
existingEntry.nodes = nodes;
if (
shouldSubscribe &&
!existingEntry.unsubscribed &&
!(forceResubscribe && existingEntry.connected === false)
) {
return;
}
```
— `replication/subscriptionManager.ts:806-821`

`existingEntry.nodes` is updated in memory, but `subscribe-to-node` is never re-posted, so the live connection keeps the original table list. The comment there explains why the early return is deliberate (re-subscribing every entry disrupts in-flight replication such as an active base copy) — the gap is that a *table-set change* is not distinguished from an ordinary node update.

## Impact

Silent, permanent, undetectable divergence for any table created after subscription on a cluster using explicit per-table subscriptions. Because the resume cursor advances past the skipped records, a later reconnect legitimately resumes beyond them — they are never re-delivered, and the retention check never sees a stale cursor, so no base-copy resync fires.

Severity is bounded by configuration: clusters on `replicateByDefault: true` (the common `databases: '*'` shape) are unaffected, because their list is an exclusion list.

## Suggested directions

1. **Refresh the table set on the live connection.** The sender already has `schemaUpdateListener = onUpdatedTable(...)` for `sendDBSchema`; the receiving end of the subscription could re-send its table list on the same trigger, or the send side could re-derive `firstNode.tables` rather than closing over the snapshot.
2. **Distinguish a table-set change from an ordinary node update** in `onDatabase`, so a genuinely new table re-posts the subscription while ordinary updates keep the current early return.
3. Independently of the fix: **a skip should not silently advance the peer's cursor.** This is the same underlying hazard as #641 — "I chose not to send you these" and "I delivered these" are indistinguishable to the subscriber. Any divergence detection built for #432 would catch this class as a whole.

## Relation to #641

Found while investigating #641. It is *not* the mechanism there (that cluster's cut was mid-stream on an existing table, and this bug would have produced zero rows rather than a partial prefix), but it is the same defect class: a send-side skip predicate that permanently advances the resume cursor over undelivered records, with no signal above `debug`.

---

*Filed by an agent (Claude Opus 5), from the #641 investigation session. Mechanism verified by reading `main` @ `2aed45a9`; not yet reproduced in a test.*

Contributor guide

Open the contributing guide

Research direction

Start in replication/replicationConnection.ts at the subscription enumeration, table filter, and skipAuditRecord paths, then trace replication/subscriptionManager.ts:806-821 and replicator.ts:775-797. Verify the live connection's table set and durable cursor behavior with a table created after subscription; done means the new table's records are delivered without silently advancing past them.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.