HarperFast / HarperFast/harper-pro
Restarted node's replication worker can hard-wedge on startup: single http worker blackholes all inbound replication while the main thread stays healthy
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 0
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 80
Description
## Summary
On a **node restart**, the node's HTTP/replication worker thread can **hard-wedge its event loop** while its main thread stays healthy. The operations API keeps answering (so `cluster_status`, `search_by_value`, etc. all respond and the node *looks* alive), but the worker that runs replication stops entirely: it accepts no new inbound WS upgrades on `:9933`, sends no pings, and emits no further log lines. When the node runs a **single** HTTP worker (`Worker Threads: 1`), this blackholes **100% of inbound replication** — peers trying to (re)connect get `WebSocket was closed before the connection was established` (code 1006). A fresh restart clears it; nothing recovers it in-process.
This is a distinct manifestation of the worker-deadlock family behind #696 (there the *whole* process goes dark and needs a container restart; here the main thread survives and only the replication worker dies).
## Evidence
**Nightly run [33239813839](https://github.com/HarperFast/harper-pro/actions/runs/33239813839)** — `Cluster Integration Tests 6/6 (Node.js v22)`, `integrationTests/cluster/removeNodeBlastRadius.test.mjs`, suite *QA-758: remove_node blast radius*. Node A (`127.0.0.2`) is restarted by test *(a, mirror)*, then test *(b)* expects an authenticated `add_node` re-join from B to reconnect. From A's restarted-instance log (`unknown-127_0_0_2-…/hdb.log`):
- A boots with **one** http worker (banner `Worker Threads: 1`), connects out to C, and accepts C's inbound subscription.
- A's **last `[http/1]` line is `07:04:28.236 … subscription to 127.0.0.2 using local id 0 starting …`** — then **zero** worker log lines for the remaining ~107 s, while `[main/0]` keeps serving the operations API (cluster_status polls; then `remove_node` at 07:06:15).
- The worker wedged at **07:04:28.24 — ~2 s *before*** B's re-join at 07:04:30.4. B opens its operation-connection to A (`527-0c:9933 4 Initializing…`) and receives **zero bytes for exactly 60 000 ms** → `Receive watchdog: no activity from undefined for 60000ms` → `WebSocket was closed before the connection was established` (1006). The re-join then times out at 105 s (`Timed out waiting for: B reconnected to A without a restart`).
- The `Reconciling … wedged … ['ws://127.0.0.2:9933']` line ~96 s later is on **B**, noticing its own dead outbound; it cannot help because **A** is the wedged node and never recovers in the window. The `{"error":"127.0.0.3 does not exist"}` (400) on the next test is a pure downstream cascade of the failed re-join (already handled by #783).
**Corroborating — run [33121734901](https://github.com/HarperFast/harper-pro/actions/runs/33121734901) (2026-08-27, shard 2, Clone_Node):** node `127.0.0.3`'s http worker goes silent after 07:18:43; peers get the identical 1006 `closed before the connection was established`; and its main thread logs `ITC broadcast … not acknowledged by worker thread(s) 2 within 30000ms` — the same "worker thread dead, main alive" shape.
## Why this is the worker-deadlock family (#696), not the subscription-setup gate (#660)
The #660 / #642 class stays **WebSocket-connected with fresh ping/pong** — the event loop is alive, only one subscription's replay never starts. Here the worker is **totally byte-silent** (no pings, no accepts; a peer sees zero bytes for 60 s). That is an event-loop block, matching #696's `/proc//wchan` evidence (the `http` thread parked on `futex_wait_queue`). The trigger window is **restart-with-replay while the worker is immediately serving an inbound subscription**; the exact native lock it parks on is not pin-able from the app logs alone.
## Detection fingerprint (for a repro harness or field diagnosis)
- The wedged node's `:9933` **listen socket accumulates a non-zero `Recv-Q`** (`ss -ltn 'sport = :9933'`) — established connections piling up unaccepted. This appears at wedge onset, no 60 s wait.
- `/proc//task/*/wchan`: a healthy node has **two** event-loop threads on `ep_poll` (main + http worker); a wedged node drops to **one** (main), with the worker thread off `ep_poll` (`futex_wait_queue` or running).
- App-level: a worker whose last replication log line is a `subscription to … using local id …` followed by silence, while the operations API stays responsive.
## Reproduction status
**Not reproduced** on Linux (20-core box) despite a targeted effort: the failing test alone (7×) and the faithful CI shape (cluster shard 6/6, `HARPER_INTEGRATION_TEST_CONCURRENCY=2`, pinned to 2 cores) both stayed green; a custom harness (8 concurrent copies at concurrency 4, `threads: { count: 1 }` to match CI, 3–6 CPU hogs on the 2 pinned cores, with real-time accept-queue + `wchan` + Node-report capture armed) ran **~85+ restart windows** and produced only **one transient** wedge (a subscription reconciler event that self-recovered) — never the fatal, non-recovering 105 s wedge. It is a rare, timing-dependent race; pinning the native lock needs a live capture (`/proc wchan` + a Node `--report-on-signal` dump, or a core dump) from an actual occurrence, most likely on a genuinely 2-core/contended runner.
## Suggested next steps
- Treat as the `main`-observed sibling of #696; a single tracked root-cause for "a replication worker can deadlock on a native lock during restart/replay" may end up owning both.
- The #696-proposed watchdog backstop does **not** cover this variant (the main thread is alive, so nothing self-restarts): recovery here needs either the worker to detect its own stall, or the node to notice a worker whose accept loop stopped (the `Recv-Q` / one-`ep_poll`-thread fingerprints above) and cycle it.
- CI capture: on a cluster-suite hang, snapshot `/proc//task/*/{comm,wchan,stat}` and `ss -ltn` for every Harper node before teardown so the next natural occurrence is diagnosable.
*Related:* #696 (whole-process self-restart deadlock — same family), #660/#642 (subscription-setup gate — the ping-alive class this is **not**), #697, and PR #783 (which fixes the misleading (c) cascade and explicitly deferred "the accept-path wedge that failed (b)" to its own issue — this is that issue).
*Attribution done as dispatch task `harperpro-qa758-rejoin-reconnect-timeout`; filed by Claude Fable (dispatch dev-agent).*
Contributor guide
Research direction
Start with integrationTests/cluster/removeNodeBlastRadius.test.mjs and the restart/rejoin path from nightly run 33239813839; compare the worker-deadlock evidence with #696. Use the listed Recv-Q, /proc task wchan, and Node report captures during a reproduction attempt. Done means identifying the deadlock or reliable stall condition and verifying recovery in the cluster test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100