HarperFast / HarperFast/harper-pro
Replicated operations have no keep-alive: any peer taking >2x replication.pingTimeout is reported failed with "Connection closed 1006" (breaks multi-node deploy_component)
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
Replicated operations (`replicateOperation` → `sendOperationToNode`) open a one-shot replication WS with **no keep-alive**, but the receive watchdog is armed on it. Nothing writes to that socket while the peer executes the operation, so the watchdog terminates it after `2 × replication.pingTimeout` (**120s** by default) and the origin reports the peer as `failed` with:
```
Error: Connection closed 1006
```
Any replicated operation whose peer-side execution exceeds ~120s therefore fails **deterministically**, even though the peer keeps executing it to completion. `deploy_component` on a component with a real `npm install` routinely crosses that line, so multi-node deploys fail intermittently — and the failure looks like a network/replication problem, which it isn't.
## Mechanism
1. `replication/replicator.ts:598` `sendOperationToNode` — `replicateOverWS(socket, {}, {})`. Options are **empty**.
2. `replication/replicationConnection.ts` — the 30s keep-alive tick is gated on `if (options.url)`, so this socket **never pings**. `sendOperation` (`:5629`) awaits `OPERATION_RESPONSE` with **no timeout**.
3. `replicationConnection.ts:2371` — `receiveWatchdog` is armed unconditionally on both ends. Window = `RECEIVE_SILENCE_THRESHOLD_MS` = `PING_TIMEOUT` = `replication.pingTimeout ?? 2 × pingInterval` = 60s default.
4. `createReceiveWatchdog` (`:1432`) re-arms once when `bytesRead` advanced since arm — the WS handshake bytes do exactly that — so the **first fire lands on the second tick, ~120s** after the socket opens.
5. Fire → no `options.connection` on this socket → `ws.terminate()` → the close handler (`:4721`) rejects every `awaitingResponse` entry with `` `Connection closed ${reason} ${code}` `` → empty reason + 1006, hence the double space in the user-visible message.
6. `replicateOperation` (`:795`) catches the rejection into `{status: 'failed', reason}`, and core's `deployComponent` (harper `components/operations.js:566-579`) turns any failed peer into a fatal 500.
Both ends are silent, so the peer's own watchdog is racing the same 120s clock; either side terminating produces the identical message.
## Evidence
Internal DevRel cluster (`prod.devrel-agents.harperfabric.com`, 2 nodes), CI deploy log:
```
16:41:09.978 prepare…
16:42:19.534 prepare done <- origin install took 70s
16:42:19.534 replicate…
16:44:19.556 replicate done
16:44:19.556 restart…
16:44:19.556 restart done
16:44:19.557 error: Component '…' was deployed on the origin node but failed to replicate to
1 of 1 peer node(s): yuz-us-central1-a-1… (Error: Connection closed 1006)
```
`replicate…` → error = **120.022s**. That is the 2 × 60s watchdog to the millisecond, not a restart race or a WAN event. The origin's own install took 70s, so the peer (which additionally waits on the payload blob before installing) sits right on top of the 120s ceiling — which is why this recurs on nearly every build for this repo.
## Secondary problem: a closed connection is reported as `failed`, not `unknown`
The peer's `OPERATION_REQUEST` handler (`replicationConnection.ts:2919`) calls `server.operation(...)` with **no abort signal**. Terminating the socket does not cancel the peer's work — it installs, loads, restarts, then tries to `ws.send` a response into a dead socket. So in the common case the component **is** deployed on the peer and the origin reports it as failed.
That makes the error actively misleading: operators can't tell it apart from a peer that really did fail (e.g. a `deployment_timeout` blob wait that expired at its own 120s), and `deploy_component`'s failure path treats the ambiguous case as fatal.
## Proposed fix
1. **Keep-alive on operation connections** — have `sendOperationToNode` pass a real keep-alive into `replicateOverWS` (e.g. `{ url: getNodeURL(node) }` or an explicit `keepAlive: true` that enables the ping tick). Pongs then feed both watchdogs, long operations survive, and a genuinely dead peer is still detected on the same clock rather than being papered over.
2. **Report closed-connection peers as indeterminate, not failed** — a transport close after the request was accepted means "outcome unknown". For `deploy_component` specifically, the origin already has an audit trail (`hdb_deployment` / `get_deployment`) it could reconcile against instead of failing fatally.
Candidate for the patch line once (1) lands — it is a small, well-scoped change and the current behavior is a hard ceiling on every replicated operation, not just deploys.
## Workarounds today
- `ignore_replication_errors: true`, then verify the peer independently (the peer records no `hdb_deployment` row of its own — `operations.js:407-421`).
- Raise `replication.pingTimeout` cluster-wide (directly widens the watchdog window); trade-off is slower genuine dead-peer detection for data replication.
- Shrink peer-side install time so the peer finishes inside 120s.
## Context
- Slack thread: https://harperdb.slack.com/archives/C3Z2T1QAZ/p1786034994970829
- Failing CI job: https://github.com/HarperFast/Developer-Relations-Agents/actions/runs/31120253840/job/92679226568
Contributor guide
Assessment
This issue has not been assessed yet.