One component's hung install cleanup deadlocks startup indefinitely — no listener opens, nothing is logged, no timeout can release it (root cause: #2076)

> **Revised (third mechanism revision).** Earlier revisions of this issue attributed the hang to (1) the 1-hour per-spawn timeout, (2) awaiting `'close'` without `'exit'` with a grandchild holding inherited pipes, and (3) a self-owned component-preparation lock whose wait deadline is renewed forever. **All three are refuted** — see "Retracted explanations" below and the comment history for the trail. The mechanism below is verified against both the running bundled core and runtime inspector state.

## Summary

A component that enters preparation on boot and whose install spawn leaves an **unreaped child** makes **startup deadlock indefinitely**. The node is completely unavailable throughout — `harper cluster_status` reports `Harper is not running`, no listener opens, replication peers cannot connect — and nothing is logged after the first second of boot. There is no timeout that can release it.

The underlying defect is an unbounded process-group termination poll, filed separately as **#2076**. This issue covers what makes it fatal rather than merely untidy: startup is gated on component preparation, so one component can take the whole node down, silently and permanently, on any restart.

Verified on `@harperfast/harper-pro@5.2.0` as running in production; the same code shape is on `main`.

## Observed

A config declared three package-based components. One had **no lock entry** in `harper-application-lock.json` and **no installed directory**, so it entered preparation on boot. Its `package:` was an SSH git remote.

**The reason that install did not complete is undetermined, and is not required for this report.** SSH deploy-key auth *was* configured: `/ssh/` contained a `config` defining exactly the host alias used by that URL, with an `IdentityFile` pointing at a present `0600` OpenSSH private key, plus a populated `known_hosts`. That is the directory `materializeGitSSH()` consumes (`join(getConfigValue(CONFIG_PARAMS.ROOTPATH), 'ssh')`), and a legacy plaintext key of that form is explicitly supported. `git` 2.47.3 is installed. So this was **not** a missing-credentials case. (A hypothesis that `ROOTPATH` might be unset during a boot-time spawn was checked and judged unreachable — `installApplications()` initialises config before any spawn.)

After the last log line —

```
(node:1) [DEP0190] DeprecationWarning: Passing args to a child process with shell option true ...
```

— nothing more was ever emitted. State at +90 minutes:

- `harper cluster_status` → `Harper is not running.`
- `grep -c 'successfully started'` → **0**; `grep -c 'timed out after'` → **0**
- 19 threads, **all** state `S` (sleeping); CPU **0.76%**, memory flat
- the direct spawn child (`sh`) present as a **zombie**, `PPid: 1`, unreaped
- one component-preparation lock ticket, published at boot, **never released**

From an inspector session on the wedged process (read-only; main thread, pid 1, uptime 5431 s):

- `threads` global is an **empty array** — no worker threads started; `server` has no listener. The block is inside `loadRootComponents` → `installApplications()`.
- `process._getActiveRequests()` → **empty**.
- libuv census: `{async:9, timer:2, check:2, idle:1, prepare:1, pipe:2, signal:3, fs_event:2, loop:1}` — **no `process` handle**, so the `ChildProcess` handle had already closed.
- The two pipes are the process's **own** `fd:1`/`fd:2`, not a child's.
- Exactly **one persistent timer**, same address across samples, re-arming at ~16-50 ms. **No long timer of any kind.**

## Mechanism

The install spawn's `'close'` handler awaits `terminateProcessTree`, which on the SIGKILL escalation path calls:

```ts
await waitForConfirmedTermination(() => processGroupIsAlive(processGroupId));
```

and in `components/Application.ts`:

```ts
const PROCESS_TERMINATION_POLL_MS = 25;

function processGroupIsAlive(processGroupId: number): boolean {
try { process.kill(-processGroupId, 0); return true; }
catch (error: any) { return error.code === 'EPERM'; }
}

export async function waitForConfirmedTermination(isAlive, pollMs = PROCESS_TERMINATION_POLL_MS) {
while (await isAlive()) await delay(pollMs); // no deadline, no cap
}
```

A process that has exited but not been reaped still occupies a PID, so `kill(-pgid, 0)` **succeeds** and `processGroupIsAlive` returns `true` for a group whose only remaining member is a zombie. `SIGTERM`/`SIGKILL` are no-ops against an already-dead process, and `waitForConfirmedTermination` has no deadline — unlike its sibling `waitForProcessGroupExit(processGroupId, timeoutMs)`, which is bounded. So the await never completes.

This accounts for every observation: a ~25 ms re-arming poll and no long timer; the zombie child being precisely what the loop waits on; no `process` handle, because `'close'` already fired and closed it; the lock ticket unreleased because the process is still inside the critical section that holds it; sleeping threads at ~0.76% CPU; and no output, indefinitely.

Full analysis of the primitive is in **#2076**.

## Why the install path is entered at all

`components/Application.ts` skips installation only when **all three** hold:

```ts
existsSync(application.dirPath) &&
harperApplicationLock.applications[name] &&
JSON.stringify(harperApplicationLock.applications[name]) === JSON.stringify(applicationConfig)
```

A component that has never successfully installed satisfies none of them, so preparation re-runs on **every** boot. There is no "this failed before, don't gate startup on it again" state. (An empty directory satisfies `existsSync`, so a half-finished install reads as installed — a related trap.)

## Why it takes the whole node down

- `server/loadRootComponents.js` — `if (isMainThread && !process.env.HARPER_SAFE_MODE) await installApplications();`
- `server/threads/threadServer.js` — `loadRootComponents(true)` is awaited; `listenOnPorts()` is only reached after it resolves.

No listener opens until every component's preparation resolves, so one component's hung cleanup takes the entire node out rather than degrading that component.

Peers cannot distinguish this from a network fault: they report `Client network socket disconnected before secure TLS connect` and loop their subscription reconciler indefinitely, because the replication port never opens.

It is also latent — the config entry sits inert until the next restart, so the outage surfaces long after the change that caused it and gets attributed to whatever triggered the restart.

## Retracted explanations

Recorded so nobody re-treads them:

1. **"Blocks for the full 1-hour spawn timeout."** Refuted — no long timer is ever armed; the node passed 90 minutes with nothing pending that could release it.
2. **"`'close'` awaited without `'exit'`, with a grandchild holding inherited pipes."** Refuted — the libuv census shows only the process's own `fd:1`/`fd:2` pipes and no `process` handle, so no child pipes were held open. `'close'` did fire; the hang is *inside* its handler.
3. **"Waiting on its own component-preparation lock ticket, deadline renewed forever."** Refuted — `scanLiveClaims` is called with `owner.token` and filters `claim.owner?.token === ownToken`, so a process cannot block on its own claim; `blocker` would be falsy and the wait loop would break immediately. The unreleased ticket is a *symptom* of hanging inside the critical section, not its cause. (Credit to an outside review for catching this.)

## Expected

1. Fix the unbounded poll — see **#2076** (bound the wait; do not count an unreaped-dead group member as alive).
2. **Do not gate the whole node on one component's preparation.** Start without it and report the component as failed, as a failed component *load* already is.
3. **Bound component preparation independently**, so no single primitive inside it can hold boot open indefinitely.
4. **Log progress.** An operator currently sees an empty log and a dead node. "Preparing component X" / "still waiting on X after Ns" would make this diagnosable in seconds.
5. Consider recording failed installs so a known-unresolvable package does not re-block every subsequent boot.

## Workaround

`HARPER_SAFE_MODE` skips `installApplications()` entirely, so a node already wedged this way can be booted with it set. Otherwise the offending component entry must be removed from the config before restarting.

## Reproducer

1. Add a component entry whose `package:` points at a git remote whose clone will fail, with no `install:` block.
2. Ensure it has no lock entry and no installed directory.
3. Restart, and arrange for the spawn's direct child to exit without being reaped.

Expect: no log output after the shell-spawn warning, `Harper is not running` indefinitely, a ~25 ms poll as the only timer, and a zombie child.

## Related

- **#2076** — the unbounded termination poll; the actual defect.
- #2061, #2063, #2001 — same family: an unbounded or unsignalled wait with no alarm.
- #1996 — deploy-transaction ordering; also touches the install/prepare path.

Filed from a field incident; cluster, host, component and repository identifiers omitted.

Read components/Application.ts and server/loadRootComponents.js first, then follow the await in server/threads/threadServer.js that precedes listenOnPorts(). Use #2076 for the termination-poll details. Done means component preparation cannot hold startup indefinitely, failures are bounded and diagnosable, and the node's listener is not withheld silently by one component.

Written by the indexing model from the issue text.