Full branch E2E binds ordinary live jobs to a stale managed-image revision
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Summary
The complete unfiltered branch E2E run for cutover PR #9323 tested the exact CLI commit but bound ordinary live managed-runtime jobs to the older `base_sha` managed-image revision. That mixes candidate source/CLI behavior with a stale runtime and causes merged fixes to recur or be bypassed.
This must be fixed without publishing a new mutable image tag or changing dependency versions. The branch run needs an exact, content-addressed candidate runtime contract or must prove that the reused runtime contains every required candidate-owned managed-startup and policy change.
## Evidence
- Full run: https://github.com/NVIDIA/NemoClaw/actions/runs/32132319706
- Candidate commit: `774e2363ecdfed90502ebbcffa03a4ecb46fc50e`
- Ordinary live jobs received `E2E_MANAGED_IMAGE_REVISION=8cdc3c41eadf51cd817df9a0f738656d8c940be5`.
- Deep Agents Code failed because the candidate onboarding reference expected the published `linux/amd64` platform contract while the base-revision validation path supplied the older multi-architecture index:
https://github.com/NVIDIA/NemoClaw/actions/runs/32132319706/job/95696286247
- Both candidate-built protected startup jobs passed, and the protected GPU/local-inference aggregate passed:
https://github.com/NVIDIA/NemoClaw/actions/runs/32132319706/job/95702206708
- Ordinary live jobs still reproduced behavior addressed by candidate-contained fixes, including CA/runtime-policy failures and the Hermes state-volume rejection.
Representative stale-runtime signatures:
- Three jobs could not read `/run/nemoclaw/managed-startup-runtime.env` and ended with `CORPORATE_CA_PROBE_FAIL`.
- Nine Hermes jobs failed with `transaction path crosses a nested filesystem mount: /sandbox/.hermes`.
- Trusted-private MCP discovery reached zero tools in both OpenClaw and Deep Agents lanes.
## Required behavior
1. An unfiltered credential-bearing branch run must test one coherent exact candidate: CLI, managed startup/runtime content, policies, and image contract must share immutable provenance.
2. The run must not silently combine a PR candidate CLI with an older base-revision managed runtime when candidate changes affect that runtime.
3. Candidate qualification must use content-addressed artifacts and digests; it must not require publishing a new mutable tag or dependency version.
4. Mixed or missing provenance must fail before fanout with an actionable error that names both revisions.
5. The repaired path must rerun the affected DCode, CA, MCP, and Hermes-volume lanes, followed by the complete unfiltered matrix.
## Security and cleanup requirements
- Keep credential authorization restricted to trusted NVIDIA branch actors.
- Do not expose secret values in provenance, logs, or artifacts.
- Preserve fail-closed policy and runtime-contract validation.
- Preserve exact-owner cleanup for any candidate containers, networks, volumes, and retained rollback resources.
## Links
- Cutover issue: #9140
- Cutover PR: #9323
- Related prior DCode issue: #9386
- Related managed runtime fixes: #9371, #9391, #9373
Contributor guide
Research direction
Start with the full unfiltered branch E2E workflow and the base-revision validation path that sets E2E_MANAGED_IMAGE_REVISION. Compare those paths with the candidate-built protected startup jobs and inspect the affected DCode, CA, MCP, and Hermes-volume lanes. Done means provenance mismatches fail before fanout, candidate artifacts are content-addressed without mutable tags, cleanup and credential restrictions remain intact, and the affected lanes plus complete matrix pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions
- Domain
- ci-cd, infrastructure, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100