HarperFast / HarperFast/harper-pro
Freshly cloned node's workers have empty `server.nodes` for minutes — replicated operations silently reach zero peers
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
On a node that was just clone-installed into a cluster, the HTTP workers' in-memory `server.nodes` stayed **empty for at least 8 minutes** after `hdb_nodes` already held both peers and the main thread had already subscribed to the leader. Every replicated operation dispatched on those workers (`replicateOperation` in `replication/replicator.ts` fans out to `server.nodes`) therefore reached **zero peers** — silently, because an empty fan-out is indistinguishable from "no peers in the cluster".
Observed on a 2-node Fabric cluster running harper-pro **5.2.3** (leader "A", clone "B"), with two consecutive `deploy_component` calls sent to B with `replicated=true`. Both `hdb_deployment` rows record `peer_results: []`, and the component never reached A. Nothing in the code paths involved changed between 5.2.3 and `origin/main` (`replication/knownNodes.ts` untouched since the tag).
## Timeline (node B, all UTC, same day)
| Time | Event |
|---|---|
| 21:23:45 | B clone-installed; `http/1` starts |
| 21:23:55 | workers restarted to apply cloned config → `http/2` starts its replication server |
| 21:23:57 | `set_node` response from leader; main logs `Setting up subscription with leader A for node A` → main's `server.nodes` has A |
| 21:23:58 | `http/2` requests full copy of `system` from A (hdb_nodes arrives via copyApply base copy); `All databases synchronized` |
| 21:28:38–21:29:21 | **deploy #1** on `http/2`, `replicated=true` → `replicate` phase `start`→`done` in the **same millisecond**, `peer_results: []` |
| 21:29:25 | deploy restart → `http/3` starts replication server; main re-logs `Setting up subscription with leader A` |
| 21:31:13–21:31:42 | **deploy #2** on `http/3`, `replicated=true` → same: `replicate` 0 ms, `peer_results: []` |
| 21:31:55 | deploy restart → `http/4` |
| ~3 h later | probe of `http/4` (`get_analytics replicated:true` via the operations UDS, which fans out over the worker's `server.nodes`) returns rows tagged for **both** nodes → the worker now knows A |
So at least two successive workers (`http/2` started before the leader row existed, `http/3` started 90 s after `hdb_nodes` was complete) ran replicated operations against an empty `server.nodes`, while a later worker on the same node is fine.
## Evidence it was really empty (not `replicated=false`)
- The CLI invocation was `harper deploy target= project= restart=true replicated=true deployment_timeout=… install_timeout=…`. Since 110b7d3 ("Replicate operations by default") `replicateOperation` only skips on `replicated === false`, so with `true` the only way to get `peer_results: []` is `server.nodes.length === 0` — a failed send would have produced a `{status:'failed'}` entry and (post harper#1333) failed the deploy.
- Both `hdb_deployment` rows: `event_log` shows `{"phase":"replicate","status":"start"}` and `{"phase":"replicate","status":"done"}` with identical `t`; `peer_results: []`; `status: "success"`.
- `hdb_nodes` on B contained both records (leader with `isLeader: true`, clone, both with `url`) when read later; B's main thread had A from 21:23:57 per the subscription log line.
## Why this matters
Every Fabric multi-node cluster is built by cloning, and the first thing an operator does after creation is deploy a component — exactly inside this window. The deploy reports success; the component exists on one node; GTM fails over to a node that 404s. Replicated `restart_service`, `drop_component`, `set_configuration`, etc. dispatched in the same window would also silently act on one node.
## Suspected area
`replication/knownNodes.ts`: workers populate `server.nodes` via replication-server start → `monitorNodeCAs` → `subscribeToNodeUpdates` → `rebuildKnownNodes` (full `hdb_nodes` scan) plus the per-row watcher (`processNodeUpdateEvent`) and the copyApply `reload` rescan (harper-pro#489). One of these isn't taking effect on a fresh clone's workers — candidates: the scan running before the cloned `system` store is the one the worker reads, the `reload` marker not reaching workers, or the `hdb_nodes` rows written by the clone's own `cloneNode` path not firing per-row events on worker threads.
## Repro (expected)
1. Start node A; clone-install node B from it (Fabric, or `HDB_LEADER_URL` / `clone` install).
2. Within a few minutes of `All databases synchronized` on B, send `deploy_component` to B with `replicated: true` (any small package).
3. Read `system.hdb_deployment` for that deployment: `peer_results` is `[]` and `status` is `success`; A never receives the component.
4. Restart B's workers (or wait) and repeat — `peer_results` now lists A.
## Related
- harper#1333 made replication *failures* fatal; zero known peers is not a failure and slips through — companion observability issue: HarperFast/harper#2253.
- harper-pro#489 (copyApply base copy + `reload` rescan), harper-pro#460 (decode-miss reconstruct) touch the same scan.
Contributor guide
Research direction
Start by tracing worker startup through replication/knownNodes.ts, including monitorNodeCAs, subscribeToNodeUpdates, rebuildKnownNodes, processNodeUpdateEvent, and the copyApply reload rescan. Reproduce the fresh-clone sequence and inspect server.nodes alongside hdb_nodes before sending a replicated deploy. Done means workers populate their peer list before replicated operations, with peer_results listing the available node rather than silently succeeding with an empty list.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100