NVIDIA / NVIDIA/NemoClaw

ci(e2e): move heavy Docker lanes to larger GitHub-hosted runners

Open
#7,145 6 comments 0 reactions 0 assignees View on GitHub
area: ci area: e2e area: performance needs: unblock platform: container v0.0.127
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

Parent epic: #7140

### Problem

The heaviest Docker/OpenShell E2E lanes run on the standard GitHub-hosted runner class. Their concurrent image builds, dependency installation, sandbox rebuilds, and cleanup can exhaust CPU, memory, or disk headroom and make hosted-runner loss difficult to distinguish from a product failure.

Adding more standard runners would reduce queue time, but it would not give an individual heavy job more resources. These lanes need a controlled larger-runner experiment.

### Scope

- Create a dedicated GitHub-hosted larger-runner group, initially sized at 8 vCPU and 32 GB RAM with substantially more workspace disk than the standard class.
- Restrict the group to trusted NemoClaw workflows and branches according to repository security policy.
- Route only resource-heavy lanes initially:
- `common-egress-agent`;
- `rebuild-hermes`;
- `rebuild-hermes-stale-base`; and
- the heavy MCP bridge matrix entries identified by baseline telemetry.
- Set group maximum concurrency to 4–6 so the workflow does not move pressure from the VM to registries or shared services.
- Keep the runner label and eligible lane list centralized so rollout and rollback do not require broad workflow edits.
- Record queue time, execution time, peak CPU/memory/disk use, infrastructure failures, and estimated cost before and after migration.

### Acceptance criteria

- The selected lanes run on the dedicated larger-runner group and unrelated fast lanes remain on standard runners.
- Maximum concurrency is explicitly configured and documented.
- At least five representative executions per migrated lane are compared with the standard-runner baseline.
- The comparison demonstrates useful resource headroom and reports duration, reliability, queue-time, and cost impact.
- Pull requests from untrusted forks cannot execute arbitrary code on a privileged or persistent runner.
- A documented rollback returns the selected lanes to standard runners without changing test semantics.

### Follow-up threshold

If larger hosted runners still disappear without process-level failure evidence, evaluate trusted-main-only ephemeral self-hosted scale sets as a separate decision. Do not introduce persistent self-hosted runners in this issue.

Contributor guide

Open the contributing guide

Research direction

Start by locating the GitHub Actions workflow definitions for the listed E2E lanes, their current runner labels, and the repository's runner security policy. Review the existing baseline telemetry and configuration points, then verify that migrated lanes, concurrency limits, comparisons, cost reporting, and rollback meet the acceptance criteria without changing test semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions
Domain
ci-cd, devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.