hashgraph / hashgraph/solo-weaver
fix(ci): reap orphaned QEMU processes and stale VM directories left by interrupted kvm-runner jobs
- Dominant language
- Go
- Stars
- 3
- Forks
- 0
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 47
Description
## Problem
`Cleanup VM` in both KVM workflows (`zxc-uat-test.yaml:481-498`,
`zxc-integration-test.yaml:509-526`) is `if: always()`, which covers step failure and job
cancellation but **not** abnormal termination of the runner itself — a runner service restart,
a host reboot, or the job process being `SIGKILL`ed. In those cases:
- the QEMU process launched at `zxc-uat-test.yaml:226-236` keeps running, holding its guest
memory allocation (`vm-mem-mb`, default 4096) and its host SSH forward port indefinitely;
- the working directory `$RUNNER_TEMP/weaver-vm-` survives, holding a qcow2 overlay that
may have grown well past its sparse initial size.
With a single runner agent the blast radius is limited — the next job gets a fresh `JOB_ID` and a
different random port, so it usually still passes while the leak quietly consumes RAM and disk.
Once multiple agents share the host to allow concurrent PRs, leaked guests directly starve
concurrent jobs, and leaked ports shrink the 2200-2599 pool that `pick_port` draws from (#966).
There is no pre-flight cleanup and no visibility: nothing reports whether orphans exist, so the
first symptom would be unexplained OOM or disk-full failures in unrelated jobs.
## Proposed fix
Add a pre-flight reaping step to the shared `kvm-vm-up` composite action (#966), before
`Create VM working directory`, that removes orphaned QEMU processes and stale VM directories
left behind by a previous job on **this agent**.
### Identify orphans by ownership, not by age
An earlier draft of this issue proposed killing QEMU processes whose elapsed time exceeds a
threshold derived from `uat-timeout-minutes` (120) / `integration-timeout-minutes` (90). That
should be avoided. Those numbers are `workflow_call` input defaults that any caller may override,
and UAT runtimes grow naturally as scenarios are added, so the threshold would have to be
maintained in lockstep with a value defined elsewhere. Set it too low and the reaper kills a
healthy long-running job, which fails as an unexplained mid-run VM death; too high and it reaps
nothing.
No threshold is needed, because there is an exact ownership signal:
- `$RUNNER_TEMP` is `/_work/_temp` — **per-agent**, not per-host.
- Each agent runs exactly one job at a time.
- Therefore, at the start of a job, any `weaver-vm-*` directory under *this agent's*
`$RUNNER_TEMP`, and any `qemu-system-x86_64` process whose argv references a path under it, is
necessarily an orphan: this agent cannot have another job running, and the current job has not
created its own directory yet.
QEMU's argv already carries the identifier, since the VM directory appears in
`-pidfile $RUNNER_TEMP/weaver-vm-/vm.pid` and in the `-drive` / `-serial` paths. Matching
on our own `$RUNNER_TEMP` prefix makes a concurrently running job on another agent untouchable by
construction rather than by hoping its elapsed time falls under a threshold.
### Steps
1. Enumerate `qemu-system-x86_64` processes whose argv contains `${RUNNER_TEMP}/weaver-vm-`
(`ps -eo pid,args`), and terminate each with `kill`, then `kill -9`, confirming the PID is
actually gone before moving on.
2. Remove any remaining `${RUNNER_TEMP}/weaver-vm-*` directories.
3. Report what was reaped — PIDs, directory names, reclaimed size — to `$GITHUB_STEP_SUMMARY`,
including an explicit "no orphans found" so accumulation is visible rather than silent.
Order matters: an orphaned QEMU holds its qcow2 overlay open, so the disk space is not reclaimed
by `rm -rf` alone — the process has to die first.
### To verify before implementing
- **Does the runner already wipe `_temp` at job start?** If it does, step 2 is mostly redundant
and step 1 is the one that matters — including for disk, per the open-file-handle note above.
- **Do the agents have distinct install directories?** The per-agent scoping argument rests
entirely on `$RUNNER_TEMP` differing between agents. Confirm this when registering the second
agent on the host; if they were ever to share a work directory, ownership-based matching would
reap a live job's guest.
### Related robustness fix
`Cleanup VM` does `kill`, then a fixed `sleep 2`, then `kill -9`, and never confirms the process
died. It should verify termination and report failure, using the same helper as step 1 above.
### Note on scope
The "leaked ports shrink the 2200-2599 pool" concern in the Problem section above is resolved by
#966: a port held by an orphaned QEMU is now detected — either by the pre-filter or by the bind
failing — and a different port is used. Losing a few of the 400 is not observable. What remains
here is the leaked guest's **memory and disk**, which is what makes this worth doing before the
host is left running unattended with multiple agents.
## Acceptance
- [ ] A pre-flight step reports and reaps orphaned QEMU processes and stale `weaver-vm-*`
directories.
- [ ] Reaping never kills a guest belonging to a concurrently running job on the same host.
- [ ] The number of reaped orphans (including zero) appears in the job summary.
- [ ] Applied to both `zxc-uat-test.yaml` and `zxc-integration-test.yaml` without duplicating the
logic.
## Notes
Prerequisite for adding concurrent runner slots to the `kvm-runner` host. Related: #966.
Contributor guide
Research direction
Start by locating the shared `kvm-vm-up` composite action and the `Cleanup VM` steps in `zxc-uat-test.yaml:481-498` and `zxc-integration-test.yaml:509-526`. Verify whether the runner wipes `_temp` at job start and whether agents have distinct install directories, then inspect the QEMU launch at `zxc-uat-test.yaml:226-236`. Done means safe pre-flight reaping and termination reporting in both workflows without duplicated logic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, shell
- Domain
- ci-cd, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100