HarperFast / HarperFast/harper-pro

Restarted node's replication worker can hard-wedge on startup: single http worker blackholes all inbound replication while the main thread stays healthy

Open
#788 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
3
Forks
0
Avg merge
1d 21h
Merged PRs (30d)
80

Description

## Summary

On a **node restart**, the node's HTTP/replication worker thread can **hard-wedge its event loop** while its main thread stays healthy. The operations API keeps answering (so `cluster_status`, `search_by_value`, etc. all respond and the node *looks* alive), but the worker that runs replication stops entirely: it accepts no new inbound WS upgrades on `:9933`, sends no pings, and emits no further log lines. When the node runs a **single** HTTP worker (`Worker Threads: 1`), this blackholes **100% of inbound replication** — peers trying to (re)connect get `WebSocket was closed before the connection was established` (code 1006). A fresh restart clears it; nothing recovers it in-process.

This is a distinct manifestation of the worker-deadlock family behind #696 (there the *whole* process goes dark and needs a container restart; here the main thread survives and only the replication worker dies).

## Evidence

**Nightly run [33239813839](https://github.com/HarperFast/harper-pro/actions/runs/33239813839)** — `Cluster Integration Tests 6/6 (Node.js v22)`, `integrationTests/cluster/removeNodeBlastRadius.test.mjs`, suite *QA-758: remove_node blast radius*. Node A (`127.0.0.2`) is restarted by test *(a, mirror)*, then test *(b)* expects an authenticated `add_node` re-join from B to reconnect. From A's restarted-instance log (`unknown-127_0_0_2-…/hdb.log`):

- A boots with **one** http worker (banner `Worker Threads: 1`), connects out to C, and accepts C's inbound subscription.
- A's **last `[http/1]` line is `07:04:28.236 … subscription to 127.0.0.2 using local id 0 starting …`** — then **zero** worker log lines for the remaining ~107 s, while `[main/0]` keeps serving the operations API (cluster_status polls; then `remove_node` at 07:06:15).
- The worker wedged at **07:04:28.24 — ~2 s *before*** B's re-join at 07:04:30.4. B opens its operation-connection to A (`527-0c:9933 4 Initializing…`) and receives **zero bytes for exactly 60 000 ms** → `Receive watchdog: no activity from undefined for 60000ms` → `WebSocket was closed before the connection was established` (1006). The re-join then times out at 105 s (`Timed out waiting for: B reconnected to A without a restart`).
- The `Reconciling … wedged … ['ws://127.0.0.2:9933']` line ~96 s later is on **B**, noticing its own dead outbound; it cannot help because **A** is the wedged node and never recovers in the window. The `{"error":"127.0.0.3 does not exist"}` (400) on the next test is a pure downstream cascade of the failed re-join (already handled by #783).

**Corroborating — run [33121734901](https://github.com/HarperFast/harper-pro/actions/runs/33121734901) (2026-08-27, shard 2, Clone_Node):** node `127.0.0.3`'s http worker goes silent after 07:18:43; peers get the identical 1006 `closed before the connection was established`; and its main thread logs `ITC broadcast … not acknowledged by worker thread(s) 2 within 30000ms` — the same "worker thread dead, main alive" shape.

## Why this is the worker-deadlock family (#696), not the subscription-setup gate (#660)

The #660 / #642 class stays **WebSocket-connected with fresh ping/pong** — the event loop is alive, only one subscription's replay never starts. Here the worker is **totally byte-silent** (no pings, no accepts; a peer sees zero bytes for 60 s). That is an event-loop block, matching #696's `/proc//wchan` evidence (the `http` thread parked on `futex_wait_queue`). The trigger window is **restart-with-replay while the worker is immediately serving an inbound subscription**; the exact native lock it parks on is not pin-able from the app logs alone.

## Detection fingerprint (for a repro harness or field diagnosis)

- The wedged node's `:9933` **listen socket accumulates a non-zero `Recv-Q`** (`ss -ltn 'sport = :9933'`) — established connections piling up unaccepted. This appears at wedge onset, no 60 s wait.
- `/proc//task/*/wchan`: a healthy node has **two** event-loop threads on `ep_poll` (main + http worker); a wedged node drops to **one** (main), with the worker thread off `ep_poll` (`futex_wait_queue` or running).
- App-level: a worker whose last replication log line is a `subscription to … using local id …` followed by silence, while the operations API stays responsive.

## Reproduction status

**Not reproduced** on Linux (20-core box) despite a targeted effort: the failing test alone (7×) and the faithful CI shape (cluster shard 6/6, `HARPER_INTEGRATION_TEST_CONCURRENCY=2`, pinned to 2 cores) both stayed green; a custom harness (8 concurrent copies at concurrency 4, `threads: { count: 1 }` to match CI, 3–6 CPU hogs on the 2 pinned cores, with real-time accept-queue + `wchan` + Node-report capture armed) ran **~85+ restart windows** and produced only **one transient** wedge (a subscription reconciler event that self-recovered) — never the fatal, non-recovering 105 s wedge. It is a rare, timing-dependent race; pinning the native lock needs a live capture (`/proc wchan` + a Node `--report-on-signal` dump, or a core dump) from an actual occurrence, most likely on a genuinely 2-core/contended runner.

## Suggested next steps

- Treat as the `main`-observed sibling of #696; a single tracked root-cause for "a replication worker can deadlock on a native lock during restart/replay" may end up owning both.
- The #696-proposed watchdog backstop does **not** cover this variant (the main thread is alive, so nothing self-restarts): recovery here needs either the worker to detect its own stall, or the node to notice a worker whose accept loop stopped (the `Recv-Q` / one-`ep_poll`-thread fingerprints above) and cycle it.
- CI capture: on a cluster-suite hang, snapshot `/proc//task/*/{comm,wchan,stat}` and `ss -ltn` for every Harper node before teardown so the next natural occurrence is diagnosable.

*Related:* #696 (whole-process self-restart deadlock — same family), #660/#642 (subscription-setup gate — the ping-alive class this is **not**), #697, and PR #783 (which fixes the misleading (c) cascade and explicitly deferred "the accept-path wedge that failed (b)" to its own issue — this is that issue).

*Attribution done as dispatch task `harperpro-qa758-rejoin-reconnect-timeout`; filed by Claude Fable (dispatch dev-agent).*

Contributor guide

Open the contributing guide

Research direction

Start with integrationTests/cluster/removeNodeBlastRadius.test.mjs and the restart/rejoin path from nightly run 33239813839; compare the worker-deadlock evidence with #696. Use the listed Recv-Q, /proc task wchan, and Node report captures during a reproduction attempt. Done means identifying the deadlock or reliable stall condition and verifying recovery in the cluster test.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.