NVIDIA / NVIDIA/NemoClaw

Full branch E2E binds ordinary live jobs to a stale managed-image revision

Open
#9,464 0 comments 0 reactions 0 assignees View on GitHub
area: ci area: e2e area: packaging integration: dcode
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Summary

The complete unfiltered branch E2E run for cutover PR #9323 tested the exact CLI commit but bound ordinary live managed-runtime jobs to the older `base_sha` managed-image revision. That mixes candidate source/CLI behavior with a stale runtime and causes merged fixes to recur or be bypassed.

This must be fixed without publishing a new mutable image tag or changing dependency versions. The branch run needs an exact, content-addressed candidate runtime contract or must prove that the reused runtime contains every required candidate-owned managed-startup and policy change.

## Evidence

- Full run: https://github.com/NVIDIA/NemoClaw/actions/runs/32132319706
- Candidate commit: `774e2363ecdfed90502ebbcffa03a4ecb46fc50e`
- Ordinary live jobs received `E2E_MANAGED_IMAGE_REVISION=8cdc3c41eadf51cd817df9a0f738656d8c940be5`.
- Deep Agents Code failed because the candidate onboarding reference expected the published `linux/amd64` platform contract while the base-revision validation path supplied the older multi-architecture index:
https://github.com/NVIDIA/NemoClaw/actions/runs/32132319706/job/95696286247
- Both candidate-built protected startup jobs passed, and the protected GPU/local-inference aggregate passed:
https://github.com/NVIDIA/NemoClaw/actions/runs/32132319706/job/95702206708
- Ordinary live jobs still reproduced behavior addressed by candidate-contained fixes, including CA/runtime-policy failures and the Hermes state-volume rejection.

Representative stale-runtime signatures:

- Three jobs could not read `/run/nemoclaw/managed-startup-runtime.env` and ended with `CORPORATE_CA_PROBE_FAIL`.
- Nine Hermes jobs failed with `transaction path crosses a nested filesystem mount: /sandbox/.hermes`.
- Trusted-private MCP discovery reached zero tools in both OpenClaw and Deep Agents lanes.

## Required behavior

1. An unfiltered credential-bearing branch run must test one coherent exact candidate: CLI, managed startup/runtime content, policies, and image contract must share immutable provenance.
2. The run must not silently combine a PR candidate CLI with an older base-revision managed runtime when candidate changes affect that runtime.
3. Candidate qualification must use content-addressed artifacts and digests; it must not require publishing a new mutable tag or dependency version.
4. Mixed or missing provenance must fail before fanout with an actionable error that names both revisions.
5. The repaired path must rerun the affected DCode, CA, MCP, and Hermes-volume lanes, followed by the complete unfiltered matrix.

## Security and cleanup requirements

- Keep credential authorization restricted to trusted NVIDIA branch actors.
- Do not expose secret values in provenance, logs, or artifacts.
- Preserve fail-closed policy and runtime-contract validation.
- Preserve exact-owner cleanup for any candidate containers, networks, volumes, and retained rollback resources.

## Links

- Cutover issue: #9140
- Cutover PR: #9323
- Related prior DCode issue: #9386
- Related managed runtime fixes: #9371, #9391, #9373

Contributor guide

Open the contributing guide

Research direction

Start with the full unfiltered branch E2E workflow and the base-revision validation path that sets E2E_MANAGED_IMAGE_REVISION. Compare those paths with the candidate-built protected startup jobs and inspect the affected DCode, CA, MCP, and Hermes-volume lanes. Done means provenance mismatches fail before fanout, candidate artifacts are content-addressed without mutable tags, cleanup and credential restrictions remain intact, and the affected lanes plus complete matrix pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, github-actions
Domain
ci-cd, infrastructure, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.