HarperFast / HarperFast/integration-testing

Detached Harper children are orphaned permanently when the runner dies by SIGKILL/SIGHUP — reap guard only covers exit/SIGINT/SIGTERM

Open
#29 0 comments 0 reactions 1 assignee Claimed by @kriszyp View on GitHub
Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
17d 7h
Merged PRs (30d)
2

Description

## Summary

`trackHarperProcess` reaps spawned Harper children on `exit`, `SIGINT`, and `SIGTERM`. It does **not** cover the ways a test runner most often actually dies — `SIGKILL`, `SIGHUP`, or an abrupt crash — and because Harper is spawned `detached` (its own process group) the child does not die with its parent. The result is a Harper instance that survives indefinitely with `PPID 1`, still holding its listening ports.

## Evidence (observed on a dev machine, 2026-08-20)

Nine orphaned `dist/bin/harper.js` processes, all `PPID 1`, all cwd'd in one `harper-pro` worktree, ages 16–18 hours:

```
PID PPID %CPU ELAPSED
93582 1 99.6 18:15:17
94348 1 99.6 18:13:33
29069 1 98.7 16:39:01
28383 1 98.7 16:40:41
28458 1 98.5 16:40:26
2574 1 98.4 18:00:40
93587 1 98.1 18:15:13
3839 1 97.9 17:59:05
2577 1 96.3 18:00:35
```

Combined **879% CPU (~8.8 cores)**. Machine load average was 21.8 with 0.14% idle; killing these nine plus two unrelated runaway jobs took it to 6.9 / 91% idle.

They were still holding their ports, which is the part that breaks subsequent runs:

```
node 93582 ... TCP 127.0.0.7:8883 (LISTEN)
node 93582 ... TCP 127.0.0.7:1883 (LISTEN)
node 93582 ... TCP 127.0.0.7:9933 (LISTEN)
node 2574 ... TCP 127.0.0.10:9933 (LISTEN)
node 28383 ... TCP 127.0.0.15:9933 (LISTEN)
```

A separate 18-day-old orphaned mocha runner was also present on the same box, so this is not a one-off.

All nine exited cleanly on a plain `SIGTERM` — nothing was wedged at the OS level; no one had signalled them.

## Why the current guard misses

`src/harperLifecycle.ts` (`trackHarperProcess`, ~L704 on `main`):

```ts
process.once('exit', reapAll);
// SIGINT/SIGTERM don't fire 'exit'; reap, then re-raise so the runner still terminates normally.
for (const signal of ['SIGINT', 'SIGTERM'] as const) {
process.once(signal, () => {
reapAll();
process.kill(process.pid, signal);
});
}
```

Not covered:

| Runner death | Handler runs? | Why it matters |
|---|---|---|
| `SIGKILL` | **No** — uncatchable | CI job cancellation, OOM killer, `kill -9` |
| `SIGHUP` | **No** — not registered | terminal/session teardown |
| hard crash / `SIGABRT` | **No** | runner segfault or abort |
| `SIGINT` / `SIGTERM` / normal exit | Yes | already handled |

The `detached: true` spawn is correct and deliberate (it's what lets teardown signal the whole group), but it also means the child is insulated from anything that kills the runner's group. Nothing in the child notices its parent is gone.

CI cancellation is the case worth calling out: cancelled jobs are a routine occurrence in the `harper-pro` stress matrix, and cancellation lands as `SIGKILL`.

## The invariant

"A Harper child never outlives its runner" is currently maintained by enumerating parent-side signals — so every uncatchable or unregistered death is a hole. It would be more robust to enforce it from the child side, where no parent cooperation is needed.

## Suggested direction (not prescriptive)

1. **Parent-liveness watchdog in the child** — the durable fix. Have the child poll for `getppid() === 1` (or watch the parent via a pipe that closes on parent death) and self-terminate. Survives `SIGKILL` because it needs nothing from the parent.
2. **Register `SIGHUP`** alongside `SIGINT`/`SIGTERM` — cheap, closes one hole, doesn't help with `SIGKILL`.
3. **Startup sweep** — on `startHarper`, reap stale Harper processes whose parent is gone before claiming an address. Complements [#13](https://github.com/HarperFast/integration-testing/issues/13), which is the same runner-killed-mid-shard root cause applied to the loopback address pool.

(1) is the only one that closes the `SIGKILL` case; (3) is the pragmatic backstop and would also reduce the port-contention failures behind #8 and #28.

## Related

- [#13 — Loopback pool leaks addresses from runners killed mid-shard (dead-PID sweep only runs when the pool is full)](https://github.com/HarperFast/integration-testing/issues/13) — same root cause (runner killed mid-shard), different leaked resource.
- [#8 — startHarper readiness deadline and teardown loopback recycle cause intermittent failures under sharded CI](https://github.com/HarperFast/integration-testing/issues/8) and [#28 — Teardown's port-release wait probes all five fixed Harper ports unconditionally](https://github.com/HarperFast/integration-testing/issues/28) — orphans holding ports are one source of the contention these describe.

**Separately:** the reason these orphans were *hot* rather than idle is a Harper-side unbounded spin lock, filed separately against `harper`. Reaping and the spin are independent defects that compounded here.

## Version

`@harperfast/integration-testing` 0.7.1; guard is unchanged on `main`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.