HarperFast / HarperFast/harper
threads.debug: worker inspector never rebinds after a worker restart ("address already in use"), leaving workers undebuggable
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
With `threads.debug` enabled, worker threads open the inspector on `debugStartingPort + workerIndex` (`server/threads/threadServer.js`). When workers are restarted (deploy restart, `restart_service`, config reload) the replacement worker starts while the outgoing worker still holds its inspector port, so `inspector.open()` fails:
```
Starting inspector on 0.0.0.0:9230 failed: address already in use
```
The failure is caught and logged at `trace` ("Could not start debugging on port …, you may already be debugging"), the old worker then exits and releases the port, and nothing ever retries — so from the first worker restart onward **no worker has an inspector**. Only the main thread (9229, which installs `closeInspector` on exit) stays debuggable.
Observed on harper-pro 5.2.3 in Fabric containers on two separate nodes; `curl http://127.0.0.1:9230/json/version` → connection refused on both while 9229 answers. Code is unchanged on `main`.
## Expected
Workers should be debuggable for the life of the process, not just until the first restart. Either:
- close the inspector in the worker on shutdown (mirror the main thread's `closeInspector` handler for `SIGTERM`/`exit`/`beforeExit` on worker threads), and have the new worker retry `inspector.open()` a few times with a short backoff if the port is still held; or
- allocate the port from a pool that accounts for overlapping old/new workers (e.g. `startingPort + restartNumber * workerCount + workerIndex`).
Low impact on its own, but it removes the main tool the Fabric investigation runbook relies on (CDP eval / profiling / logpoints on workers).
Contributor guide
Research direction
Start with server/threads/threadServer.js and trace how worker startup opens the inspector and how shutdown/restart overlaps with replacement workers. The issue describes two possible approaches, so first determine how the existing worker lifecycle handles inspector cleanup and retries. Done means workers regain an inspector after restart and the inspector endpoint responds on its port.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100