ci(e2e): classify hosted-runner resource pressure and infrastructure loss
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
Parent epic: #7140
Related dependency: #7101 owns general secret-safe phase heartbeats and progress coverage across E2E targets. This issue owns accurate resource attribution, failure classification, and retry policy.
### Problem
Current host-memory snapshots can make healthy Linux page cache look like memory exhaustion, while a GitHub-hosted VM can disappear before cleanup, logs, or artifacts identify the cause. Without cgroup, pressure, process, and Docker evidence, maintainers cannot reliably distinguish:
- application assertion failures;
- a process or container OOM kill;
- disk or inode exhaustion;
- a stalled Docker build; and
- hosted-runner infrastructure loss.
That uncertainty encourages broad retries, which can hide deterministic regressions.
### Scope
- Extend the secret-safe heartbeat contract with accurate host measurements where available:
- `MemAvailable`, `Cached`, `SReclaimable`, swap use, and load;
- cgroup `memory.current`, peak/limit, `memory.events`, and OOM/OOM-kill counters;
- memory and I/O pressure stall information;
- top process RSS/PSS consumers;
- Docker container stats, image/build-cache usage, workspace free space, and inode availability; and
- kernel OOM evidence where the hosted environment permits access.
- Emit bounded snapshots before and after expensive image/install/rebuild phases and periodically while a phase is active.
- Produce a machine-readable terminal classification for ordinary failures: assertion, timeout, process OOM, container OOM, disk pressure, or unknown.
- Define workflow-level signatures for hosted-runner loss when the runner disappears before test cleanup can execute.
- Permit at most one retry only for confirmed infrastructure-loss signatures. Never retry assertions, deterministic command failures, policy violations, or classified OOM failures.
- Keep all streamed and uploaded evidence free of command payloads, credentials, tokens, and environment-variable values.
### Acceptance criteria
- Low raw `MemFree` alone is never classified as OOM.
- Tests cover classification of application failure, timeout, cgroup OOM, disk exhaustion, runner loss, and ambiguous failure.
- An ordinary assertion receives zero automatic retries.
- A confirmed hosted-runner-loss result receives no more than one retry and links the two attempts for diagnosis.
- The original failure evidence remains available when a retry succeeds.
- Representative `rebuild-hermes` and other heavy lanes emit enough evidence to identify the largest host/container memory consumers and Docker disk use.
- The implementation interoperates with #7101 rather than creating a second progress-heartbeat framework.
### Non-goals
- Increasing test timeouts.
- Retrying failures that lack a positive infrastructure-loss classification.
- Treating larger runners as a substitute for diagnostics.
Contributor guide
Assessment
This issue has not been assessed yet.