ci(e2e): improve reliability and observability of long-running runners
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Summary
Long-running Docker/OpenShell E2E jobs can lose their GitHub-hosted runner without producing a failed test step, cleanup output, downloadable logs, or diagnostic artifacts. This slows release validation and makes infrastructure failures difficult to distinguish from product failures.
## Evidence
Final-main E2E run: https://github.com/NVIDIA/NemoClaw/actions/runs/29631712998
Commit: `4fe17e1693e2bb122c017ecaabb4bc97ec10d39c`
Three jobs ended with `failure` after their live-test step remained `in_progress` with no conclusion:
- [`common-egress-agent`](https://github.com/NVIDIA/NemoClaw/actions/runs/29631712998/job/88046601227)
- [`rebuild-hermes`](https://github.com/NVIDIA/NemoClaw/actions/runs/29631712998/job/88046601294)
- [`rebuild-hermes-stale-base`](https://github.com/NVIDIA/NemoClaw/actions/runs/29631712998/job/88046601329)
For all three jobs:
- no failed step was recorded
- artifact upload and cleanup never started
- no diagnostic artifact was created
- no downloadable job log was available
- each job ran on a separate standard `ubuntu-latest` runner
These are long Docker/OpenShell lanes:
- `common-egress-agent`: 120-minute timeout
- `rebuild-hermes`: 90-minute timeout
- `rebuild-hermes-stale-base`: 90-minute timeout
A separate failure in the same run selected a stale base image because final E2E started before the corrected image publication completed. That dependency race is distinct from the runner-loss failures, but it should be removed from the release-validation path as part of making the workflow deterministic.
## Proposed direction
1. Route long Docker/OpenShell E2E lanes to a dedicated larger-runner group, initially around 8 vCPU / 32 GB RAM with materially more disk.
2. Bound heavy-runner concurrency, initially around 4–6 jobs.
3. Record periodic CPU, memory, disk, Docker-storage, and runner-heartbeat telemetry.
4. Split or checkpoint long tests so phase diagnostics are uploaded before the entire live-test step completes.
5. Add one tightly classified retry for runner-infrastructure loss; never retry assertion failures automatically.
6. Gate final-main E2E on successful publication of all required base images for the exact commit.
7. If larger hosted runners remain unreliable, evaluate ephemeral trusted-main-only self-hosted runners with externally forwarded runner diagnostics.
Adding more standard runners alone may reduce queue time, but it will not prevent an assigned runner from disappearing mid-job. The runner class, disk headroom, observability, and failure classification need to be addressed together.
## Acceptance criteria
- Long-running release E2E lanes use a documented runner class with adequate CPU, memory, and disk.
- Heavy-lane concurrency is bounded.
- Resource telemetry and phase-level diagnostics survive ordinary test failures.
- Runner-loss failures are distinguishable from product or test failures.
- Infrastructure retries occur at most once and never mask assertion failures.
- Final-main E2E starts only after required base images for the exact commit are published.
Contributor guide
Assessment
This issue has not been assessed yet.