HarperFast / HarperFast/harper
Worker thread exits cleanly (code 0) before reporting ready during concurrent multi-worker startup, aborting the whole node
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
### Symptom
Recurring Linux Integration Tests failure on `main` — currently the top non-Windows CI failure (red in 3 of the last 5 runs):
```
Error: Worker (index 1) exited with code 0 before reporting ready
at Worker.onExit (server/threads/socketRouter.ts:100:11)
```
immediately followed in *sibling* workers by a `Database not open` cascade, after which every in-flight test in the shard is cancelled.
Occurrences (same shape, different shards, suites, and Node versions — 22 / 24 / 26.5):
- 2026-08-25 https://github.com/HarperFast/harper/actions/runs/32812674585 — 3/6 (Node 22), deleteUpdateRace.test.ts
- 2026-08-24 https://github.com/HarperFast/harper/actions/runs/32732580586 — 1/6 (uWS), delete-update-race-consistency.test.ts; 3/6 (Node 22) blob.test.mjs
- 2026-08-24 https://github.com/HarperFast/harper/actions/runs/32741812476 — 3/6 (Node 26.5.0), record-caching-cross-worker.test.ts
- 2026-08-24 https://github.com/HarperFast/harper/actions/runs/32701165774 — 3/6 (Node 24), scheduled nightly
The affected suites are all multi-worker (`threads.count: 4`), RocksDB-engine, high-concurrency startup tests.
### Root cause
A Harper worker thread that has not yet posted `child_started` owns **no ref'd libuv handle**:
- `manageThreads.addPort()` explicitly `unref()`s the worker's `parentPort` (and sibling ports); `threadServer.js` only `.ref()`s it *after* `loadRootComponents()` resolves.
- Component file watchers are chokidar `persistent: false`; the resource-report and monitoring timers are `unref()`d.
So during `loadRootComponents()`, any `await` whose completion arrives through a source that does not hold the event loop lets the loop drain — and a worker whose loop drains exits **cleanly with code 0**, before the ready handshake. `createWorkerReadyPromise` then rejects, `startHTTPThreads`' `Promise.all(workersReady)` rejects, and `bin/run.ts`'s catch calls `process.exit(1)` — the whole node aborts during boot.
The natural trigger for exactly this window is rocksdb-js: it delivers **cross-thread lock-release wakes** (`DBDescriptor::lockEnqueueCallback`, used by `Store.tryLock(key, onUnlocked)`) and **parked-commit retry wakes** (coordinated-retry `RETRY_NOW` in `transaction.cpp`) through threadsafe functions that are `napi_unref_threadsafe_function`'d at creation. A pre-ready worker parked on a peer worker's lock — e.g. the cross-thread `store.tryLock(componentDirectory)` in `componentLoader.ts`'s `symlinkHarperModule()`, whose non-acquired branch additionally **clears the only ref'd timer** before waiting — contributes nothing to its own loop's liveness and can exit before the wake arrives. Four workers concurrently loading the same components at first boot make that contention structural; whether the loser happens to have other in-flight ref'd work at the drain instant is the flake coin-flip.
Post-ready workers park on the same locks constantly but survive, because the ref'd `parentPort` (and bound listeners) hold the loop — which is why this only ever surfaces as "exited … before reporting ready".
The mechanism reproduces standalone: a worker with an unref'd `parentPort` awaiting a completion behind an unref'd handle exits code 0 at the drain instant. The full-instance drain did not reproduce locally on a 20-core dev box (constant startup ITC traffic keeps reviving the loop there); it reproduces on loaded 4-vCPU CI runners.
### Cause vs. effect
The `Database not open` cascade in sibling workers is an **effect**, not a second defect: main's `process.exit(1)` tears down stores while sibling workers are still mid component-load (the stack in the log: `processGraphQLSchema → table → openAuditStore → RocksTransactionLogStore`).
### Fix
1. `threadServer.js`: hold `parentPort.ref()` for the entire pre-ready window (SHUTDOWN still unrefs for graceful exit) — a worker must report ready, throw, or be terminated; never silently drain.
2. `componentLoader.ts` `symlinkHarperModule()`: the lock waiter keeps a ref'd bounded timer instead of clearing it (also fixes an unbounded wait if the lock holder dies, and removes the winner's stale 10s timer that could `unlock()` a lock it no longer owned).
A companion rocksdb-js issue covers the unref'd threadsafe-function wakes themselves (pending cross-thread lock/commit-retry wakes should keep the waiting thread's loop alive).
Related: #1827 (startup hang with no output) may share the `symlinkHarperModule` unbounded-lock-wait surface fixed here.
Contributor guide
Assessment
This issue has not been assessed yet.