NVIDIA / NVIDIA/NemoClaw

ci(e2e): investigate the protected runtime critical path

Open
#9,539 0 comments 0 reactions 1 assignee Claimed by @rsliter View on GitHub
area: ci area: e2e v0.0.127
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Problem

The protected managed-image lane dominated the critical path of a full E2E push run.

Source run: https://github.com/NVIDIA/NemoClaw/actions/runs/32178707220
Source commit SHA: `b2d1ce52a716444b083f6e3b8ed8bace1cba3240`

| Phase | Duration |
| --- | ---: |
| Complete workflow | 2h 16m 53s |
| Dispatch to first job | 31m 03s |
| Dispatch to protected job start | 1h 06m 08s |
| Protected job | 57m 32s |
| Build three protected sandbox images | 17m 13s |
| Qualify GPU, local inference, rollback, and cleanup | 36m 34s |
| Protected job completion to scorecard start | 12m 39s |

Five commits landed on `main` while this exact-SHA run executed. Reducing this critical path would reduce candidate drift without changing E2E coverage.

Run https://github.com/NVIDIA/NemoClaw/actions/runs/32191102944 at SHA `5f9269683a1ddfb6ad5403bc91398297cced995d` provides a second component sample. Its base-image gate failed, so this is not a second full GPU and provider qualification sample.

| Phase | Duration |
| --- | ---: |
| Complete workflow | 1h 45m 55s |
| Dispatch to first job | 1h 08m 15s |
| Protected startup, linux/arm64 | 19m 04s |
| Protected startup, linux/amd64 | 22m 29s |
| amd64 all-agent image build | 20m 23s |
| amd64 direct managed-image contracts | 43s |
| Last executable test completion to scorecard start | 5m 29s |
| Scorecard | 17s |

All protected startup and Jetson jobs passed. Five newer commits landed while this exact-SHA run executed. This sample shows that image construction and runner handoffs can dominate even when the direct runtime contracts are short.

The current build script calls `build_agent` sequentially for OpenClaw, Hermes, and LangChain Deep Agents Code. The live qualification then evaluates every agent sequentially for Ollama, vLLM, NVIDIA NIM, and rollback. Shared GPU, provider, registry, sandbox, and cleanup state can make some parallel execution unsafe, so the investigation must measure those boundaries before proposing a change.

## Investigation

- Record per-agent image-build time, cache result, provider startup time, per-agent qualification time, rollback time, and cleanup time.
- Collect at least five executions of this lane when available and bind every sample to its exact commit SHA and runner class.
- Determine which build operations can execute concurrently without sharing mutable BuildKit, registry, or output state.
- Determine whether any qualification operations can execute concurrently without GPU contention, provider contention, sandbox-name collision, or cleanup interference.
- Identify repeated setup that can be reused without weakening exact-image, rollback, cleanup, or credential-boundary verification.
- Quantify expected wall-clock savings and runner-cost changes for each viable option.

## Acceptance criteria

- The investigation identifies the measured critical path for each sampled execution.
- The proposed plan preserves the same agents, providers, GPU behavior, rollback behavior, cleanup behavior, and exact-image verification.
- The plan names each shared mutable resource and explains why proposed concurrency is isolated.
- The plan includes a rollback and a way to compare runtime and failure rate before and after the change.
- The issue does not add a live E2E test, broaden a retry, or change release judgment.

Parent epic: #9159

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.