HarperFast / HarperFast/harper-pro

replication.databases: "*" subscribes databases with no replicable tables, causing an endless subscribe/timeout/close loop (~4,300/node/day)

Open Beginner friendly
#685 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

> **Suggested priority: P3** — pure noise/waste, no data at risk, and the fix is a one-line
> predicate change. Prioritized at all because ~44% of a quiet node's error-level lines are this
> loop, which materially slowed diagnosis of the P0/P1 replication issues (#683, #684).

**Area:** replication · **Line refs:** `v5.2.1` · **Seen on:** 5.2.1 (4-node production cluster)

## Problem

With `replication.databases: "*"`, every local database is subscribed for replication — including a
database that has **no replicable tables at all**. The peer correctly reports that it has nothing
registered for that database, subscription setup never completes, and #642's setup watchdog closes
the socket so the subscriber can retry. The retry hits the same wall, so the cycle repeats forever.

Measured on an idle-ish production cluster, per node, over one hour:

| signal | count/hour |
| --- | --- |
| `Timed out waiting for database subscription setup for ` | 170–173 |
| `No database named "" was declared and registered` | 148–149 |
| `Disconnected from wss://:9933 (db: "")` | 181–188 |

That is ~4,300 connect/timeout/close cycles per node per day, indefinitely. On a node not otherwise
busy, these were **173 of 393 total `[error]` lines (44%)**.

The database in question is an application database whose single table is declared
`@table(database: "coordination", replicate: false)` — deliberately node-local (it backs a per-node
SharedArrayBuffer). Nothing about it should ever be replicated, and the application is not doing
anything unusual: it simply exists as a database, and `"*"` means "subscribe to everything".

## Cause

`replication/replicator.ts`, `forReplicatedDatabase`:

```js
if (
options?.databases === undefined ||
options.databases === '*' ||
options.databases.includes(databaseName) ||
options.databases.some?.((dbConfig) => dbConfig.name === databaseName) ||
!database
)
callback(database, databaseName, true);
else if (hasExplicitlyReplicatedTable(databaseName)) callback(database, databaseName, false);
```

The predicate that would exclude this database already exists — `hasExplicitlyReplicatedTable()` —
but it is only consulted on the `else` branch. The wildcard/default path subscribes unconditionally.

## Suggested fix

Consult `hasExplicitlyReplicatedTable(databaseName)` on the wildcard/default path too: a database
with zero replicable tables should not be subscribed regardless of how `replication.databases` is
configured. (An existing local database with no replicated tables is knowable locally, so this
needs no protocol change. Keep the `!database` case as-is — that is the isLeader/bootstrap path for
databases that do not exist yet.)

Alternatively/additionally, treat "peer reports the database is not registered" as a terminal
condition for that (peer, database) subscription rather than a retryable one, with backoff instead
of an unbounded tight retry.

## Why it matters beyond waste

The volume of error-level output actively hinders diagnosis. While investigating #683 on this
cluster, I initially attributed two `page_cache` reconnects to this churn purely from temporal
proximity; the counts refuted it (2,090 coordination disconnects vs 2 page_cache disconnects over
the same 12h). Filling error logs at ~4,300/day/node with a condition that is both expected and
permanent makes real replication faults materially harder to find.

## Workaround, and why it is not a good one

Setting `replication.databases` to an explicit list excluding the database takes the `else` branch
and fixes it. But unlisted databases then get `replicateByDefault: false`, so any table relying on
the default rather than an explicit `replicate: true` would silently stop replicating — including
internal databases if the operator's list is incomplete. That is a large footgun to hand someone
whose only goal is to stop a log flood.

Contributor guide

Open the contributing guide

Research direction

Start in replication/replicator.ts at forReplicatedDatabase and inspect how the wildcard/default predicate uses hasExplicitlyReplicatedTable(). Verify the change with a local database containing no replicable tables, while preserving the !database bootstrap path; done means that database is not subscribed under wildcard configuration and the retry loop stops.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
82/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.