HarperFast / HarperFast/integration-testing

Address retirement on port-release timeout is one-shot for the process lifetime; with a 1-address pool the run deadlocks in `getNextAvailableLoopbackAddress` even though retirement is meant to be a parking action

Open
#27 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
17d 7h
Merged PRs (30d)
2

Description

`teardownHarper` waits `DEFAULT_PORT_RELEASE_TIMEOUT_MS` (default 5s) for all fixed Harper
ports to free, and on expiry it does not release the loopback address
(`dist/harperLifecycle.js:618-629` in 0.7.1): the slot stays parked under the current
process's PID and the warning says "The slot will be reclaimed when this process exits."
There is no later re-check. Retirement is permanent for the process lifetime, however briefly
the ports were actually held.

That interacts badly with two facts:

- Under `--isolation=none` (harper-integration-test-run) every suite in the run shares one
process, so "reclaimed when this process exits" means "never, for the rest of this run".
- On stock macOS only 127.0.0.1 is bindable (127.0.0.2+ need aliases), so a practical local
pool is one address (`HARPER_INTEGRATION_TEST_LOOPBACK_POOL_START=1`, `COUNT=1`).

After one retirement, the next suite's `startHarper` enters
`getNextAvailableLoopbackAddress` and spins forever: the free-slot scan finds nothing and
`removeDeadProcessesFromPool` (`dist/loopbackAddressPool.js:414-425`) cannot reclaim a slot
whose holder PID is the caller's own live process. The wait is unbounded and silent.

Measured sequence, one macOS host, 2026-08-14, three consecutive
`npm run test:integration` invocations of the same two-file suite:

| Run (local time) | `PORT_RELEASE_TIMEOUT_MS` | Suite 1 | Then |
| --- | --- | --- | --- |
| 15:42 | 5000 (default) | green, 15.2s total | wait expired, warning logged, address retired |
| 15:53 | 5000 (default) | green, 13.4s total | same |
| 16:05 | 60000 (override) | green, 68.1s total | 60s wait also expired; suite 2 never acquired an address; the harness had to be killed externally and `$TMPDIR/harper-integration-test-loopback-pool.json` reset by hand |

Raising the timeout 12x changed only the stall length, which separates the two defects: why
the ports were held is its own issue (on this host an unrelated resident MQTT broker held
1883, filed separately), but the one-shot retirement is what turns any expiry into a wedged
run. In the case the code comments actually contemplate (a slow Harper child that exits a few
seconds after the deadline), the ports free and the address stays retired anyway; that case
follows directly from the code path (no re-check exists) and is expected rather than
separately measured here.

A second, related defect: the pool persists holder PIDs in
`$TMPDIR/harper-integration-test-loopback-pool.json` across processes
(`dist/loopbackAddressPool.js:23`). A run killed with SIGKILL leaves its PID parked; after
the 16:05 run was killed, the pool file still carried that run's slot state and was reset to
`[]` by hand before anything else ran (measured). What happens without that reset is a code
read, not a measurement: reclaim runs only when the pool is exhausted and tests liveness
with `process.kill(pid, 0)`, so a dead run's slot is reclaimed eventually, unless the OS has
reused the PID for any live process, in which case the slot stays parked indefinitely and
later runs are poisoned by a run that no longer exists.

Suggested fixes, independently useful:

- On each acquisition attempt inside the same process, re-probe addresses this process
previously retired instead of skipping them for the process lifetime. The conflict canary
in `getNextAvailableLoopbackAddress` already exists to make handing out a still-held
address safe; a freed address should return to service on the next probe.
- Stamp pool entries with the holder's process start time (or a timestamp plus a nonce) so
`removeDeadProcessesFromPool` can distinguish a dead run from a reused PID.
- Name the still-bound ports in the teardown warning (today it prints none), so an operator
can see in one line whether the holder is a Harper child or an unrelated service.

## Reproduction

1. macOS with no loopback aliases; `HARPER_INTEGRATION_TEST_LOOPBACK_POOL_START=1`,
`HARPER_INTEGRATION_TEST_LOOPBACK_POOL_COUNT=1`.
2. Ensure any listener occupies one fixed Harper port on 127.0.0.1 for the duration (e.g.
`nc -l 127.0.0.1 1883 &`), so the port-release wait must expire.
3. Run two Harper suites via `harper-integration-test-run --isolation=none`.
4. Observed: suite 1 passes; teardown logs
`Harper ports on 127.0.0.1 still in use after teardown (5000ms); NOT recycling the address`;
suite 2 blocks forever in `getNextAvailableLoopbackAddress` with no output. Kill the run;
the pool JSON still holds the dead PID.

## Measured on

| Component | Version |
| --- | --- |
| @harperfast/integration-testing | 0.7.1 (`dist/harperLifecycle.js`, `dist/loopbackAddressPool.js`) |
| harper (system under test) | 5.2.1 |
| Node | v24.16.0 |
| OS | macOS 26.5.2 (arm64, Darwin 25.5.0), stock loopback (127.0.0.1 only) |
| suites | datadog-agent-binary `bbeb99a` integration tests, `--isolation=none` |

---

Contributor guide

Open the contributing guide

Research direction

Start with teardownHarper in dist/harperLifecycle.js:618-629 and trace getNextAvailableLoopbackAddress and removeDeadProcessesFromPool in dist/loopbackAddressPool.js:414-425. Reproduce with a one-address pool, --isolation=none, and a listener occupying a fixed port. Done means consecutive suites no longer wedge after a timeout, and the persisted pool state remains safe across killed or restarted processes.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.