HarperFast / HarperFast/harper

Worker threads never close their inspector: replaced workers fail to open the debug port with EADDRINUSE, logged only at trace

Open
#2,108 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

Worker threads never close their V8 inspector, so after any worker generation is replaced the
inspector ports stay bound and the new workers fail to open theirs. The failure is caught and logged
at **trace**, so in production it is completely invisible — worker-thread debuggability is silently
and permanently lost until a full process restart.

## Cause

`core/server/threads/threadServer.js` registers the close handler only for the main thread:

```js
if (isMainThread) {
port = env.get(terms.CONFIG_PARAMS.THREADS_DEBUG_PORT) ?? 9229;
const closeInspector = () => {
try { require('inspector').close(); } catch (error) { ... }
};
for (const signal of ['SIGINT', 'SIGTERM', 'SIGQUIT', 'exit']) {
process.on(signal, closeInspector);
}
} else {
// workers: no closeInspector registered anywhere
const startingPort = env.get(terms.CONFIG_PARAMS.THREADS_DEBUG_STARTINGPORT);
if (startingPort && getWorkerIndex() >= 0) port = startingPort + getWorkerIndex();
}

try {
require('inspector').open(port, host, waitForDebugger);
} catch (error) {
harperLogger.trace(`Could not start debugging on port ${port} ...`); // <-- trace
}
```

Worker N binds `THREADS_DEBUG_STARTINGPORT + N`. When that worker is replaced (config reload, worker
restart, worker crash-and-respawn) it exits without closing the inspector, the port stays bound, and
the replacement worker's `inspector.open()` throws `address already in use`.

## Impact

Observed **96 occurrences** on a single production node. Every worker that had been replaced was
undebuggable, with no error, warning, or metric to indicate it — the only way to find out is to try
to attach and get nothing, or to grep at trace level.

This matters specifically because the main thread is often not where the interesting state lives. We
just spent hours on a snapshot-retention investigation (harper-pro#662) where the holder could have
been on any of 17 threads; we could only enumerate all of them after a full process restart
reopened every inspector.

## Suggested fix

Register `closeInspector` for workers too. Worker threads don't receive process signals the same
way, so the natural hooks are the worker's own exit path — `process.on('exit')` inside the worker, or
whatever teardown already runs when a worker generation is retired.

Also worth raising the log level: a failure to open the inspector port is a diagnostic capability
silently disappearing, which is at least `warn`, not `trace`.

## Detection

Host-side port probes lie, because docker-proxy accepts on the published port regardless of whether
anything inside the container is listening. Check inside the container's netns:

```sh
docker logs 2>&1 | grep -c 'address already in use'
# or, listening sockets in the Harper process's netns:
awk 'NR>1 && $4=="0A" {split($2,a,":"); printf "%d\n", strtonum("0x" a[2])}' /proc//net/tcp
```

Environment: `harperfast/harper-pro:5.1.26`, Node v24.18.1, `THREADS_DEBUG_STARTINGPORT=9230`,
`THREADS_DEBUG_HOST=0.0.0.0`, 16 http workers.

Contributor guide

Open the contributing guide

Research direction

Start in core/server/threads/threadServer.js, comparing the main-thread signal handlers with the worker startup and teardown path. Reproduce a worker replacement with inspector debugging enabled, then verify that the replacement can bind its assigned port and that inspector-open failures are visible at the intended log level.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, devtools
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.