NVIDIA / NVIDIA/NemoClaw

ci(e2e): improve reliability and observability of long-running runners

Open
#7,140 5 comments 0 reactions 1 assignee Claimed by @apurvvkumaria View on GitHub
area: ci area: e2e v0.0.127
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Summary

Long-running Docker/OpenShell E2E jobs can lose their GitHub-hosted runner without producing a failed test step, cleanup output, downloadable logs, or diagnostic artifacts. This slows release validation and makes infrastructure failures difficult to distinguish from product failures.

## Evidence

Final-main E2E run: https://github.com/NVIDIA/NemoClaw/actions/runs/29631712998
Commit: `4fe17e1693e2bb122c017ecaabb4bc97ec10d39c`

Three jobs ended with `failure` after their live-test step remained `in_progress` with no conclusion:

- [`common-egress-agent`](https://github.com/NVIDIA/NemoClaw/actions/runs/29631712998/job/88046601227)
- [`rebuild-hermes`](https://github.com/NVIDIA/NemoClaw/actions/runs/29631712998/job/88046601294)
- [`rebuild-hermes-stale-base`](https://github.com/NVIDIA/NemoClaw/actions/runs/29631712998/job/88046601329)

For all three jobs:

- no failed step was recorded
- artifact upload and cleanup never started
- no diagnostic artifact was created
- no downloadable job log was available
- each job ran on a separate standard `ubuntu-latest` runner

These are long Docker/OpenShell lanes:

- `common-egress-agent`: 120-minute timeout
- `rebuild-hermes`: 90-minute timeout
- `rebuild-hermes-stale-base`: 90-minute timeout

A separate failure in the same run selected a stale base image because final E2E started before the corrected image publication completed. That dependency race is distinct from the runner-loss failures, but it should be removed from the release-validation path as part of making the workflow deterministic.

## Proposed direction

1. Route long Docker/OpenShell E2E lanes to a dedicated larger-runner group, initially around 8 vCPU / 32 GB RAM with materially more disk.
2. Bound heavy-runner concurrency, initially around 4–6 jobs.
3. Record periodic CPU, memory, disk, Docker-storage, and runner-heartbeat telemetry.
4. Split or checkpoint long tests so phase diagnostics are uploaded before the entire live-test step completes.
5. Add one tightly classified retry for runner-infrastructure loss; never retry assertion failures automatically.
6. Gate final-main E2E on successful publication of all required base images for the exact commit.
7. If larger hosted runners remain unreliable, evaluate ephemeral trusted-main-only self-hosted runners with externally forwarded runner diagnostics.

Adding more standard runners alone may reduce queue time, but it will not prevent an assigned runner from disappearing mid-job. The runner class, disk headroom, observability, and failure classification need to be addressed together.

## Acceptance criteria

- Long-running release E2E lanes use a documented runner class with adequate CPU, memory, and disk.
- Heavy-lane concurrency is bounded.
- Resource telemetry and phase-level diagnostics survive ordinary test failures.
- Runner-loss failures are distinguishable from product or test failures.
- Infrastructure retries occur at most once and never mask assertion failures.
- Final-main E2E starts only after required base images for the exact commit are published.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.