Per-container operations hang indefinitely (create stuck at [6/6]; teardown never completes after workload exit); hung ops cascade and can drop a custom network's gateway until the hung client is killed (1.0.0)

## Environment

- `container` 1.0.0 (Homebrew `container` keg 1.0.0_1)
- macOS 26 (Tahoe), Darwin 25.5.0, Mac Studio M3 Ultra (28 cores, 28% avg load during incidents)
- Custom NAT network: `container network create --subnet 192.168.72.0/24 runners` (helper cmdline: `container-network-vmnet start --id runners --service-identifier com.apple.container.network.container-network-vmnet.runners --mode nat --subnet 192.168.72.0/24`)
- Workload: ephemeral Linux containers used as GitHub Actions JIT runners — up to 10 concurrent, mostly 1–5 min lifetimes, 100–300 launches/hour in bursts, launches deliberately spaced ≥10 s apart (pacing did not prevent anything below)

## Summary

Under sustained short-lived-container churn we observe three related behaviors, all consistent with per-container XPC operations having **no timeout** and one hung operation blocking others behind it (head-of-line). Evidence timestamps are from 2026-07-02 (UTC).

### A. `container run` hangs at `[6/6] Starting container` indefinitely; the record is never created

- CLI spinner observed at `[6/6] Starting container [28m 52s]` … `[49m 38s]`.
- In one 95-minute window (17:31–19:05Z): **283 `container run` invocations → only 70 containers ever started their workload**; at peak **63 concurrent hung `container run` CLI processes**, while other containers on the same network booted normally seconds apart from the hung ones.
- A hung create never materializes a record (`container ls --all` does not show the name), so `container stop` / `container delete` are no-ops for it and the CLI waits forever. Killing the `container run` process is the only way to release it.

### B. Containers whose workload exited never tear down

- The containerized process (GitHub runner agent) completes and exits cleanly ~45 s after boot; `container ls` still shows `running` 65+ minutes later, and the per-container `container-runtime-linux` process persists.
- `container stop` on such a container sometimes works (returns in ~5 s and the blocked `container run` client then exits); at other times — concurrent with shape A — `container stop` itself hangs.

### C. While hung per-container processes exist, the custom network's gateway IP can vanish from the host; killing the hung processes restores it within ~1 second

Timeline:

- 01:03:30–31Z — three containers launched; their jobs completed 42–46 s later; the containers never tore down (shape B).
- 01:05:04Z — gateway `192.168.72.1` is no longer present on any host interface (`ifconfig`); guests lose TCP egress (in-container `nc -z 1.1.1.1 443` fails; workloads report lost network).
- 02:08:30Z and 02:09:37Z — the two hung containers are force-stopped (`container stop` worked here, ~5 s each).
- 02:09:44Z — **one second after the second cleanup — the gateway address is back** and subsequent launches get IPs + egress normally. Neither `container-apiserver` nor `container-network-vmnet` was restarted at any point; the network object was never deleted or recreated.
- The same drop→kill→instant-restore pattern reproduced three more times the same day. In one instance, killing **one** hung client caused **five** queued `container run` creates (hung 60+ min) to all return within one second — classic head-of-line release.
- The daemons never crash: the unified log shows only healthy XPC connection churn for `container-apiserver` in these windows, and `container-network-vmnet` logs **nothing at default level** across a 65-minute outage.

### Frequency

Some episode of A/B/C occurs every ~10–40 minutes under our churn. It reproduces with launches spaced 10–22 s apart, so a launch burst is not required.

## Ruled out

- **IP exhaustion** — /24 subnet, ≤ ~15 addresses in use; 1.0.0's connection-as-lease allocation (#1545) verifiably frees IPs when the hung processes are killed.
- **Host pressure** — 27–68% CPU, no memory pressure; a parallel Virtualization.framework workload (Tart macOS VMs) on the same host is unaffected throughout.
- **Guest images** — the same images boot in seconds once the hung client processes are killed.

## Related issues

#857 / #1162 (XPC timeout at containerCreate/ping — matches shape A's front edge), #861 (`system stop` hangs on the vmnet helper), #1321 (`vmnet_network_ref` lifecycle), #679 (force-quit via per-container `launchctl kill` — the workaround class we converged on), #1389 (TCP silently dropped with gateway present). What this report adds: the completed-workload-never-exits shape, the gateway-vanishes-while-hung-processes-exist ↔ restored-in-~1s-on-kill correlation, head-of-line release of queued creates, and rate/scale data from a CI fleet.

## Workaround that works reliably (no daemon restarts, no reboot)

Reconcile container state against external job state, then kill only per-container processes: TERM/KILL the hung `container run` client; if the record still shows `running`, TERM/KILL its `container-runtime-linux … --uuid ` process. This releases queued operations immediately, restores the gateway when it had dropped, and frees the IP. `container-apiserver` / `container-network-vmnet` never need to be touched.

## Asks

1. Timeouts on per-container XPC operations — the create/start path behind `[6/6]`, and the exit/teardown notification path — so a hung operation fails instead of blocking forever (related: your own CI timeout was raised to 75 min in #1663).
2. Per-operation logging (or queue visibility) in `container-network-vmnet` — it logs nothing at default level even during an hour-long outage, so the blocking operation is unidentifiable from outside.
3. Clarification whether one hung attach/detach blocking the helper (and dropping the gateway address from the host) is a known failure mode.

Happy to provide extended unified-log captures, run instrumented builds, or gather a sysdiagnose during the next episode — it recurs frequently enough that turnaround would be fast.

Start with the per-container XPC create/start and exit/teardown paths in container-apiserver, then inspect container-network-vmnet's attach/detach handling. Reproduce the described short-lived-container churn while collecting the mentioned unified logs; done should mean bounded operations, no queued head-of-line hangs, and logging that identifies a blocker.

Written by the indexing model from the issue text.