HarperFast / HarperFast/harper

threads.debug: worker inspector never rebinds after a worker restart ("address already in use"), leaving workers undebuggable

Open
#2,254 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

With `threads.debug` enabled, worker threads open the inspector on `debugStartingPort + workerIndex` (`server/threads/threadServer.js`). When workers are restarted (deploy restart, `restart_service`, config reload) the replacement worker starts while the outgoing worker still holds its inspector port, so `inspector.open()` fails:

```
Starting inspector on 0.0.0.0:9230 failed: address already in use
```

The failure is caught and logged at `trace` ("Could not start debugging on port …, you may already be debugging"), the old worker then exits and releases the port, and nothing ever retries — so from the first worker restart onward **no worker has an inspector**. Only the main thread (9229, which installs `closeInspector` on exit) stays debuggable.

Observed on harper-pro 5.2.3 in Fabric containers on two separate nodes; `curl http://127.0.0.1:9230/json/version` → connection refused on both while 9229 answers. Code is unchanged on `main`.

## Expected

Workers should be debuggable for the life of the process, not just until the first restart. Either:

- close the inspector in the worker on shutdown (mirror the main thread's `closeInspector` handler for `SIGTERM`/`exit`/`beforeExit` on worker threads), and have the new worker retry `inspector.open()` a few times with a short backoff if the port is still held; or
- allocate the port from a pool that accounts for overlapping old/new workers (e.g. `startingPort + restartNumber * workerCount + workerIndex`).

Low impact on its own, but it removes the main tool the Fabric investigation runbook relies on (CDP eval / profiling / logpoints on workers).

Contributor guide

Open the contributing guide

Research direction

Start with server/threads/threadServer.js and trace how worker startup opens the inspector and how shutdown/restart overlaps with replacement workers. The issue describes two possible approaches, so first determine how the existing worker lifecycle handles inspector cleanup and retries. Done means workers regain an inspector after restart and the inspector endpoint responds on its port.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.