HarperFast / HarperFast/harper

Integration-test Harper servers have no parent-death watchdog and survive their test runner indefinitely

Open
#2,603 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

Harper servers spawned by the integration-test harness have no parent-death watchdog. When the
test runner dies — timeout, crash, `SIGKILL`, an agent being cancelled — the servers it started
survive, get reparented to `init`/`systemd --user`, and run **forever**.

## Evidence

Six leaked `dist/bin/harper.js` processes on one dev box, all reparented to PPID 1-equivalent
(`systemd --user`), ages 50 minutes to 2 hours, together consuming **~15 of 20 cores**:

| PID | Age | Node addr | CPU |
|---|---|---|---|
| 2859868 | 2h04 | 127.0.0.10 | ~2.5 cores |
| 2861463 | 2h04 | 127.0.0.24 | ~2.6 cores |
| 2953736 | 1h58 | 127.0.0.27 | ~1.7 cores |
| 3558979 | 54m | 127.0.0.7 | ~2.5 cores |
| 3559555 | 54m | 127.0.0.2 | ~2.6 cores |
| 3589118 | 50m | 127.0.0.11 | ~2.6 cores |

All six came from a single worktree's test run, had zero established connections, and exited
cleanly on `SIGTERM` — nothing was holding them open. Machine load average dropped from **24 to
1.5** once they were killed.

## Why there is no guard today

`HARPER_PARENT_PROCESS_PID` looks like it would cover this, but it is a **restart handshake**,
not a watchdog. `bin/run.ts:128-143` reads it once at startup to wait for a *previous* process
to finish compacting RocksDB, then deletes it from the environment:

```js
if (process.env.HARPER_PARENT_PROCESS_PID) {
const prevProcessPid = parseInt(process.env.HARPER_PARENT_PROCESS_PID);
delete process.env.HARPER_PARENT_PROCESS_PID;
if (isProcessRunning(prevProcessPid)) { /* wait up to 15s for it to exit */ }
}
```

`resources/databases.ts:926-931` reads the same variable and only logs. Neither establishes any
ongoing liveness relationship, so once the parent dies there is nothing to notice.

## Impact

Contained to developer machines and CI runners — this is test-harness lifecycle, not shipped
server behavior. But it is unbounded and the trigger is structural (any runner that does not
exit cleanly), and two things make it worse than a stray idle process:

- The leaked nodes bind fixed loopback addresses and ports (`127.0.0.x:9925/9926/9933/1883/8883`),
so they squat the ports a later local run needs and can make subsequent runs fail or flake.
- They are not idle. Each one burned ~2.5 cores continuously, via #2602.

CI containers are torn down wholesale, so this is mostly invisible there and shows up on
long-lived dev machines.

## Suggested fix

Give spawned test servers a parent-liveness check — either periodically poll the runner pid and
`process.exit()` when it disappears, or use a pipe/IPC channel whose close event terminates the
child. A harness-side `after`/teardown hook is worth having too, but on its own it cannot cover
the case that actually leaks here, which is the runner dying without running teardown.

Contributor guide

Open the contributing guide

Research direction

Start with the HARPER_PARENT_PROCESS_PID handling in bin/run.ts:128-143 and resources/databases.ts:926-931, then trace how the integration-test harness spawns Harper servers. Add a parent-liveness mechanism so servers terminate when the runner disappears, and verify that forced runner termination leaves no server processes or occupied test ports.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.