HarperFast / HarperFast/harper
Worker threads never close their inspector: replaced workers fail to open the debug port with EADDRINUSE, logged only at trace
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
Worker threads never close their V8 inspector, so after any worker generation is replaced the
inspector ports stay bound and the new workers fail to open theirs. The failure is caught and logged
at **trace**, so in production it is completely invisible — worker-thread debuggability is silently
and permanently lost until a full process restart.
## Cause
`core/server/threads/threadServer.js` registers the close handler only for the main thread:
```js
if (isMainThread) {
port = env.get(terms.CONFIG_PARAMS.THREADS_DEBUG_PORT) ?? 9229;
const closeInspector = () => {
try { require('inspector').close(); } catch (error) { ... }
};
for (const signal of ['SIGINT', 'SIGTERM', 'SIGQUIT', 'exit']) {
process.on(signal, closeInspector);
}
} else {
// workers: no closeInspector registered anywhere
const startingPort = env.get(terms.CONFIG_PARAMS.THREADS_DEBUG_STARTINGPORT);
if (startingPort && getWorkerIndex() >= 0) port = startingPort + getWorkerIndex();
}
try {
require('inspector').open(port, host, waitForDebugger);
} catch (error) {
harperLogger.trace(`Could not start debugging on port ${port} ...`); // <-- trace
}
```
Worker N binds `THREADS_DEBUG_STARTINGPORT + N`. When that worker is replaced (config reload, worker
restart, worker crash-and-respawn) it exits without closing the inspector, the port stays bound, and
the replacement worker's `inspector.open()` throws `address already in use`.
## Impact
Observed **96 occurrences** on a single production node. Every worker that had been replaced was
undebuggable, with no error, warning, or metric to indicate it — the only way to find out is to try
to attach and get nothing, or to grep at trace level.
This matters specifically because the main thread is often not where the interesting state lives. We
just spent hours on a snapshot-retention investigation (harper-pro#662) where the holder could have
been on any of 17 threads; we could only enumerate all of them after a full process restart
reopened every inspector.
## Suggested fix
Register `closeInspector` for workers too. Worker threads don't receive process signals the same
way, so the natural hooks are the worker's own exit path — `process.on('exit')` inside the worker, or
whatever teardown already runs when a worker generation is retired.
Also worth raising the log level: a failure to open the inspector port is a diagnostic capability
silently disappearing, which is at least `warn`, not `trace`.
## Detection
Host-side port probes lie, because docker-proxy accepts on the published port regardless of whether
anything inside the container is listening. Check inside the container's netns:
```sh
docker logs 2>&1 | grep -c 'address already in use'
# or, listening sockets in the Harper process's netns:
awk 'NR>1 && $4=="0A" {split($2,a,":"); printf "%d\n", strtonum("0x" a[2])}' /proc//net/tcp
```
Environment: `harperfast/harper-pro:5.1.26`, Node v24.18.1, `THREADS_DEBUG_STARTINGPORT=9230`,
`THREADS_DEBUG_HOST=0.0.0.0`, 16 http workers.
Contributor guide
Research direction
Start in core/server/threads/threadServer.js, comparing the main-thread signal handlers with the worker startup and teardown path. Reproduce a worker replacement with inspector debugging enabled, then verify that the replacement can bind its assigned port and that inspector-open failures are visible at the intended log level.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, devtools
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100