HarperFast / HarperFast/harper-pro
replication.databases: "*" subscribes databases with no replicable tables, causing an endless subscribe/timeout/close loop (~4,300/node/day)
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
> **Suggested priority: P3** — pure noise/waste, no data at risk, and the fix is a one-line
> predicate change. Prioritized at all because ~44% of a quiet node's error-level lines are this
> loop, which materially slowed diagnosis of the P0/P1 replication issues (#683, #684).
**Area:** replication · **Line refs:** `v5.2.1` · **Seen on:** 5.2.1 (4-node production cluster)
## Problem
With `replication.databases: "*"`, every local database is subscribed for replication — including a
database that has **no replicable tables at all**. The peer correctly reports that it has nothing
registered for that database, subscription setup never completes, and #642's setup watchdog closes
the socket so the subscriber can retry. The retry hits the same wall, so the cycle repeats forever.
Measured on an idle-ish production cluster, per node, over one hour:
| signal | count/hour |
| --- | --- |
| `Timed out waiting for database subscription setup for ` | 170–173 |
| `No database named "" was declared and registered` | 148–149 |
| `Disconnected from wss://:9933 (db: "")` | 181–188 |
That is ~4,300 connect/timeout/close cycles per node per day, indefinitely. On a node not otherwise
busy, these were **173 of 393 total `[error]` lines (44%)**.
The database in question is an application database whose single table is declared
`@table(database: "coordination", replicate: false)` — deliberately node-local (it backs a per-node
SharedArrayBuffer). Nothing about it should ever be replicated, and the application is not doing
anything unusual: it simply exists as a database, and `"*"` means "subscribe to everything".
## Cause
`replication/replicator.ts`, `forReplicatedDatabase`:
```js
if (
options?.databases === undefined ||
options.databases === '*' ||
options.databases.includes(databaseName) ||
options.databases.some?.((dbConfig) => dbConfig.name === databaseName) ||
!database
)
callback(database, databaseName, true);
else if (hasExplicitlyReplicatedTable(databaseName)) callback(database, databaseName, false);
```
The predicate that would exclude this database already exists — `hasExplicitlyReplicatedTable()` —
but it is only consulted on the `else` branch. The wildcard/default path subscribes unconditionally.
## Suggested fix
Consult `hasExplicitlyReplicatedTable(databaseName)` on the wildcard/default path too: a database
with zero replicable tables should not be subscribed regardless of how `replication.databases` is
configured. (An existing local database with no replicated tables is knowable locally, so this
needs no protocol change. Keep the `!database` case as-is — that is the isLeader/bootstrap path for
databases that do not exist yet.)
Alternatively/additionally, treat "peer reports the database is not registered" as a terminal
condition for that (peer, database) subscription rather than a retryable one, with backoff instead
of an unbounded tight retry.
## Why it matters beyond waste
The volume of error-level output actively hinders diagnosis. While investigating #683 on this
cluster, I initially attributed two `page_cache` reconnects to this churn purely from temporal
proximity; the counts refuted it (2,090 coordination disconnects vs 2 page_cache disconnects over
the same 12h). Filling error logs at ~4,300/day/node with a condition that is both expected and
permanent makes real replication faults materially harder to find.
## Workaround, and why it is not a good one
Setting `replication.databases` to an explicit list excluding the database takes the `else` branch
and fixes it. But unlisted databases then get `replicateByDefault: false`, so any table relying on
the default rather than an explicit `replicate: true` would silently stop replicating — including
internal databases if the operator's list is incomplete. That is a large footgun to hand someone
whose only goal is to stop a log flood.
Contributor guide
Research direction
Start in replication/replicator.ts at forReplicatedDatabase and inspect how the wildcard/default predicate uses hasExplicitlyReplicatedTable(). Verify the change with a local database containing no replicable tables, while preserving the !database bootstrap path; done means that database is not subscribed under wildcard configuration and the retry loop stops.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 82/100