HarperFast / HarperFast/harper-pro

Freshly cloned node's workers have empty `server.nodes` for minutes — replicated operations silently reach zero peers

Open
#746 0 comments 0 reactions 0 assignees View on GitHub
area:clustering area:replication bug
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

On a node that was just clone-installed into a cluster, the HTTP workers' in-memory `server.nodes` stayed **empty for at least 8 minutes** after `hdb_nodes` already held both peers and the main thread had already subscribed to the leader. Every replicated operation dispatched on those workers (`replicateOperation` in `replication/replicator.ts` fans out to `server.nodes`) therefore reached **zero peers** — silently, because an empty fan-out is indistinguishable from "no peers in the cluster".

Observed on a 2-node Fabric cluster running harper-pro **5.2.3** (leader "A", clone "B"), with two consecutive `deploy_component` calls sent to B with `replicated=true`. Both `hdb_deployment` rows record `peer_results: []`, and the component never reached A. Nothing in the code paths involved changed between 5.2.3 and `origin/main` (`replication/knownNodes.ts` untouched since the tag).

## Timeline (node B, all UTC, same day)

| Time | Event |
|---|---|
| 21:23:45 | B clone-installed; `http/1` starts |
| 21:23:55 | workers restarted to apply cloned config → `http/2` starts its replication server |
| 21:23:57 | `set_node` response from leader; main logs `Setting up subscription with leader A for node A` → main's `server.nodes` has A |
| 21:23:58 | `http/2` requests full copy of `system` from A (hdb_nodes arrives via copyApply base copy); `All databases synchronized` |
| 21:28:38–21:29:21 | **deploy #1** on `http/2`, `replicated=true` → `replicate` phase `start`→`done` in the **same millisecond**, `peer_results: []` |
| 21:29:25 | deploy restart → `http/3` starts replication server; main re-logs `Setting up subscription with leader A` |
| 21:31:13–21:31:42 | **deploy #2** on `http/3`, `replicated=true` → same: `replicate` 0 ms, `peer_results: []` |
| 21:31:55 | deploy restart → `http/4` |
| ~3 h later | probe of `http/4` (`get_analytics replicated:true` via the operations UDS, which fans out over the worker's `server.nodes`) returns rows tagged for **both** nodes → the worker now knows A |

So at least two successive workers (`http/2` started before the leader row existed, `http/3` started 90 s after `hdb_nodes` was complete) ran replicated operations against an empty `server.nodes`, while a later worker on the same node is fine.

## Evidence it was really empty (not `replicated=false`)

- The CLI invocation was `harper deploy target= project= restart=true replicated=true deployment_timeout=… install_timeout=…`. Since 110b7d3 ("Replicate operations by default") `replicateOperation` only skips on `replicated === false`, so with `true` the only way to get `peer_results: []` is `server.nodes.length === 0` — a failed send would have produced a `{status:'failed'}` entry and (post harper#1333) failed the deploy.
- Both `hdb_deployment` rows: `event_log` shows `{"phase":"replicate","status":"start"}` and `{"phase":"replicate","status":"done"}` with identical `t`; `peer_results: []`; `status: "success"`.
- `hdb_nodes` on B contained both records (leader with `isLeader: true`, clone, both with `url`) when read later; B's main thread had A from 21:23:57 per the subscription log line.

## Why this matters

Every Fabric multi-node cluster is built by cloning, and the first thing an operator does after creation is deploy a component — exactly inside this window. The deploy reports success; the component exists on one node; GTM fails over to a node that 404s. Replicated `restart_service`, `drop_component`, `set_configuration`, etc. dispatched in the same window would also silently act on one node.

## Suspected area

`replication/knownNodes.ts`: workers populate `server.nodes` via replication-server start → `monitorNodeCAs` → `subscribeToNodeUpdates` → `rebuildKnownNodes` (full `hdb_nodes` scan) plus the per-row watcher (`processNodeUpdateEvent`) and the copyApply `reload` rescan (harper-pro#489). One of these isn't taking effect on a fresh clone's workers — candidates: the scan running before the cloned `system` store is the one the worker reads, the `reload` marker not reaching workers, or the `hdb_nodes` rows written by the clone's own `cloneNode` path not firing per-row events on worker threads.

## Repro (expected)

1. Start node A; clone-install node B from it (Fabric, or `HDB_LEADER_URL` / `clone` install).
2. Within a few minutes of `All databases synchronized` on B, send `deploy_component` to B with `replicated: true` (any small package).
3. Read `system.hdb_deployment` for that deployment: `peer_results` is `[]` and `status` is `success`; A never receives the component.
4. Restart B's workers (or wait) and repeat — `peer_results` now lists A.

## Related

- harper#1333 made replication *failures* fatal; zero known peers is not a failure and slips through — companion observability issue: HarperFast/harper#2253.
- harper-pro#489 (copyApply base copy + `reload` rescan), harper-pro#460 (decode-miss reconstruct) touch the same scan.

Contributor guide

Open the contributing guide

Research direction

Start by tracing worker startup through replication/knownNodes.ts, including monitorNodeCAs, subscribeToNodeUpdates, rebuildKnownNodes, processNodeUpdateEvent, and the copyApply reload rescan. Reproduce the fresh-clone sequence and inspect server.nodes alongside hdb_nodes before sending a replicated deploy. Done means workers populate their peer list before replicated operations, with peer_results listing the available node rather than silently succeeding with an empty list.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.