hashgraph / hashgraph/solo-weaver

feat(ci): add a kvm-runner host capacity probe and per-job resource sampler to size additional runner slots

Open
#968 0 comments 0 reactions 0 assignees View on GitHub
github-actions Improvement Pending Triage
Dominant language
Go
Stars
3
Forks
0
Avg merge
3d 2h
Merged PRs (30d)
46

Description

## Problem

`kvm-runner` is a **single** statically-registered self-hosted runner agent (runner name
`kvm-runner`, group `Default`), unlike `solo-linux-medium` which is an autoscaling ARC scale set
(group `solo - primary`). A runner agent executes one job at a time, so all five KVM-dependent
jobs from every open PR serialize through one slot.

Measured on run `30879804373` (one PR's full cycle):

| Job | Queued | Ran |
|---|---|---|
| UAT Compat | 7m 13s | 11m 52s |
| UAT Core | 17m 3s | 17m 28s |
| UAT Purge Storage | 36m 7s | 13m 50s |
| UAT Plugin Preset Upgrade | 49m 58s | 11m 29s |
| Integration Tests | 24m 25s | 58m 24s |

That is **~2h15m of queueing against ~1h53m of actual work**. The jobs demonstrably interleave
across PRs: run `30878576912`'s UAT Core occupied 05:33-05:50, exactly the gap where run
`30879804373`'s UAT Core sat queued.

The fix is more agents on the `kvm-runner` label, but there is no way to size that safely today.
Nobody knows the host's CPU/RAM/disk ceiling, how many agents are already registered on it, or
what a single job consumes at peak. The host is hand-provisioned and its spec is recorded nowhere:
an org-wide code search for `kvm-runner` returns only this repo's two workflow files, and neither
`hashgraph/infrastructure` nor `hashgraph/runner-images` contains any QEMU/KVM runner
provisioning.

## Proposed fix

Two complementary measurements. They have to be separate, because with only one slot a standalone
probe job queues *behind* the real job and therefore always observes an idle host.

### 1. Capacity probe — new `workflow_dispatch` workflow on `kvm-runner`

Reports the ceiling to `$GITHUB_STEP_SUMMARY`. Every command `|| true`-guarded so one missing tool
cannot abort the probe.

- **Host identity and exact size** — cloud metadata gives the machine type directly, removing all
guesswork about vCPU/RAM (and settles whether the host is even on GCP):
`curl -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/machine-type`
plus `/instance/name`, `/instance/zone`, `/project/project-id`. Fall back to AWS IMDSv2 / Azure
IMDS, then `hostname` and `dmidecode -s system-product-name`.
- **Compute/memory** — `nproc`, `lscpu`, `free -h`, `MemTotal`/`MemAvailable`, `swapon --show`,
`/proc/loadavg`.
- **Existing slot count** — `systemctl list-units 'actions.runner.*' --all`, and the agent version
from `/.runner`, so new agents can match it. This is the single most important number.
- **Disk** — `df -h` for `/`, `/var/lib/weaver`, `/home/github-runner`, `$RUNNER_TEMP`;
`du -sh /var/lib/weaver/images`.
- **Competing load** — `docker ps`, since the Squid (`3128`), Go module proxy (`8081`), and
registry mirror (`5050`) services the jobs tunnel into consume RAM unavailable to guests if they
live on this host.
- **Leak counters** — orphaned `qemu-system` processes, stale `weaver-vm-*` dirs, and stuck ports
in the 2200-2599 range, feeding #966 and the orphan-reaping work.

### 2. Per-job demand sampler — temporary instrumentation in `zxc-uat-test.yaml`

A background loop started after `Start QEMU VM (KVM)` writing every 10s to
`$VM_DIR/host-usage.tsv`: timestamp, `/proc/loadavg`, used/available MB, total QEMU RSS, and free
space on `$RUNNER_TEMP`. A teardown step ordered **before** `Cleanup VM` (whose
`rm -rf "${VM_DIR}"` would otherwise destroy the data) kills the sampler and appends peak/mean to
the job summary, plus `du -sh "$VM_DIR/vm-disk.qcow2"` for the overlay's real growth.

Trigger note: a brand-new `workflow_dispatch` workflow cannot be dispatched from a feature branch,
but `flow-test-uat.yaml` is already on `main` and calls the reusable workflow by relative path, so
`gh workflow run flow-test-uat.yaml --ref -f uat-scenario=compat` picks up the branch's
edited copy.

### Sizing arithmetic once both numbers exist

Slots are bounded by the tightest of RAM (per job: `vm-mem-mb` 4096 + ~0.5 GB QEMU overhead;
guests run a real kubeadm cluster and genuinely touch their allocation), CPU (`vm-cpus` 3 — but
use *measured* peak, because the 10-minute SSH-ready gate and in-guest Kubernetes readiness probes
turn CPU starvation into flaky failures rather than slow passes), disk (measured overlay growth ×
slots), and I/O (all overlays share one backing image on one device; concurrent `kubeadm init` is
disk-heavy).

## Acceptance

- [ ] A dispatchable workflow reports host identity/machine type, core count, memory, disk
headroom, and the registered `actions.runner.*` unit count.
- [ ] A UAT job reports peak and mean host CPU/memory plus real qcow2 overlay growth for one job.
- [ ] The two figures together yield a defensible maximum slot count for the host.
- [ ] Leak counters are included, so #966 and orphan reaping can be assessed from the same run.

## Notes

The sampler is an instrument, not a shipping feature — it can stay on a throwaway branch. The
capacity probe is worth keeping permanently as an ops tool. Registering the additional agents
themselves requires repo or org **admin** rights
(`POST /repos/hashgraph/solo-weaver/actions/runners/registration-token`), which is a separate
follow-up. Related: #966.

Contributor guide

Open the contributing guide

Research direction

Start by reading flow-test-uat.yaml to see how the reusable workflow is invoked, then inspect zxc-uat-test.yaml around Start QEMU VM and Cleanup VM. Use the documented gh workflow run command to exercise the branch and check the step summaries and host-usage.tsv output. Done means the dispatchable probe reports the listed host and leak data, and one UAT run reports peak/mean usage and qcow2 growth for slot sizing.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions, kubernetes, linux, shell, yaml
Domain
ci-cd, devops, infrastructure, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.