HarperFast / HarperFast/harper-pro
Replication: tables created after a subscription are silently never replicated when replicateByDefault is false, with the resume cursor advancing over their records
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
When a peer subscribes with `replicateByDefault: false`, the table list it sends is an **inclusion** list snapshotted at subscription time. A table created *after* that subscription is established is therefore not in the list, and every audit record for it is silently skipped on the send side — **with the peer's durable resume cursor advancing over those records the whole time**.
The subscription is not re-posted when a table is added, so this does not self-heal. It persists for the life of the connection and is invisible: no error, no warning, `cluster_status` healthy, and the cursor stays "valid" so no base-copy resync is triggered.
Line references are against `main` @ `2aed45a9` (v5.2.1).
## Mechanism
**1. The list is a snapshot, and it is an inclusion list when `replicateByDefault` is false.**
The subscription request enumerates the local table registry once (`replication/replicationConnection.ts:5225-5253`). When a node has explicit `subscriptions` configured, `replicateByDefault` is forced off:
```js
replicateByDefault = false; // now turn off the default replication because it was an explicit list of subscriptions
```
— `replication/replicationConnection.ts:5246`
`replicateByDefault: false` also arises via `forReplicatedDatabase` for a database that is not in `options.databases` but contains explicitly-replicated tables (`replication/replicator.ts:797`).
**2. The send side gates every record on that snapshot.**
```js
const tableToTableEntry = (table) => {
if (
table &&
(firstNode.replicateByDefault
? !firstNode.tables.includes(table.tableName)
: firstNode.tables.includes(table.tableName))
) {
return { table };
}
};
```
— `replication/replicationConnection.ts:3538`
`firstNode` is `nodeSubscriptions[0]`, captured when the subscription is handled and never refreshed. With `replicateByDefault: true` the list is an *exclusion* list, so new tables replicate by default and everything is fine. With it false, a table absent from the snapshot can never match.
**3. A non-match is a silent skip that advances the cursor.**
`tableToTableEntry` returns `undefined`, so the record falls into:
```js
logger.debug?.('Not subscribed to table', tableId);
return skipAuditRecord();
```
— `replication/replicationConnection.ts:3605-3606`
`skipAuditRecord()` (`:3723`) arms a 300 ms timer that sends `[SEQUENCE_ID_UPDATE, currentSequenceId]` (`:3732`). The receiver converts that into an `end_txn` (`:3112`) and core's apply loop persists it as the durable resume cursor (`core/resources/Table.ts:757`, applied at `:856`). The skipped records are now permanently behind the cursor.
**4. Adding a table does not re-post the subscription.**
`onUpdatedTable` does reach the subscription manager — `forEachReplicatedDatabase` registers `onUpdatedTable((Table) => forReplicatedDatabase(Table.databaseName, ...))` (`replication/replicator.ts:775`) — which calls `onDatabase(databaseName, tablesReplicateByDefault, forceResubscribe = false)`.
But for an already-subscribed database that path returns early:
```js
if (existingEntry) {
worker = existingEntry.worker;
existingEntry.nodes = nodes;
if (
shouldSubscribe &&
!existingEntry.unsubscribed &&
!(forceResubscribe && existingEntry.connected === false)
) {
return;
}
```
— `replication/subscriptionManager.ts:806-821`
`existingEntry.nodes` is updated in memory, but `subscribe-to-node` is never re-posted, so the live connection keeps the original table list. The comment there explains why the early return is deliberate (re-subscribing every entry disrupts in-flight replication such as an active base copy) — the gap is that a *table-set change* is not distinguished from an ordinary node update.
## Impact
Silent, permanent, undetectable divergence for any table created after subscription on a cluster using explicit per-table subscriptions. Because the resume cursor advances past the skipped records, a later reconnect legitimately resumes beyond them — they are never re-delivered, and the retention check never sees a stale cursor, so no base-copy resync fires.
Severity is bounded by configuration: clusters on `replicateByDefault: true` (the common `databases: '*'` shape) are unaffected, because their list is an exclusion list.
## Suggested directions
1. **Refresh the table set on the live connection.** The sender already has `schemaUpdateListener = onUpdatedTable(...)` for `sendDBSchema`; the receiving end of the subscription could re-send its table list on the same trigger, or the send side could re-derive `firstNode.tables` rather than closing over the snapshot.
2. **Distinguish a table-set change from an ordinary node update** in `onDatabase`, so a genuinely new table re-posts the subscription while ordinary updates keep the current early return.
3. Independently of the fix: **a skip should not silently advance the peer's cursor.** This is the same underlying hazard as #641 — "I chose not to send you these" and "I delivered these" are indistinguishable to the subscriber. Any divergence detection built for #432 would catch this class as a whole.
## Relation to #641
Found while investigating #641. It is *not* the mechanism there (that cluster's cut was mid-stream on an existing table, and this bug would have produced zero rows rather than a partial prefix), but it is the same defect class: a send-side skip predicate that permanently advances the resume cursor over undelivered records, with no signal above `debug`.
---
*Filed by an agent (Claude Opus 5), from the #641 investigation session. Mechanism verified by reading `main` @ `2aed45a9`; not yet reproduced in a test.*
Contributor guide
Research direction
Start in replication/replicationConnection.ts at the subscription enumeration, table filter, and skipAuditRecord paths, then trace replication/subscriptionManager.ts:806-821 and replicator.ts:775-797. Verify the live connection's table set and durable cursor behavior with a table created after subscription; done means the new table's records are delivered without silently advancing past them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100