NVIDIA / NVIDIA/NemoClaw

[CI] Managed image catalog and CLI build identity resolve different releases for the same commit when a tag lands mid-run

Open
#11,282 0 comments 0 reactions 0 assignees View on GitHub
area: ci area: e2e area: packaging
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Investigation Summary

- `PR exact all-agent managed runtime activation` failed with `managed image catalog contract for 'openclaw' belongs to 'v0.0.120-83-g624480df1', not 'v0.0.121-7-g624480df1'`.
- Both strings describe the **same commit** `624480df1`; only the nearest tag differs, so the candidate images and the candidate CLI were stamped with different releases within one workflow run.
- The run's own log shows the tag arriving mid-run: `* [new tag] v0.0.121 -> v0.0.121`. `getVersion()` resolves from `git describe --tags --match "v*"` at build time, so whichever step builds before the tag is fetched gets the older description.
- Correlates exactly with tag ancestry, not with PR content: PR #11166 (`624480df1`, contains the `v0.0.121` tag commit `673f815e0`) failed; PR #11266 (`676221b91`, does not contain it) passed the same check.
- Re-running only the failed jobs fails again because it reuses the images built in attempt 1. A full workflow re-run passes, because the images are then rebuilt with the tag present.

## Description

`v0.0.121` is tagged at `673f815e0` ("docs: prepare v0.0.121 documentation (#11243)"). Once a PR branch contains that commit, the managed-image publication workflow can stamp its candidate images and its candidate CLI with two different releases for one commit, and the activation gate then refuses the images as belonging to another release:

```text
[non-interactive] Agent: OpenClaw
Error: Sandbox workload preparation failed: managed image catalog contract for 'openclaw'
belongs to 'v0.0.120-83-g624480df1', not 'v0.0.121-7-g624480df1'
```

The refusal itself is correct -- `preparation.ts` is right to reject a catalog that belongs to a different release. The defect is upstream: the two build steps do not observe the same tag state, so they disagree about what release the commit is.

**Expected:** every step in one workflow run derives the same release string for the same commit, so a newly pushed tag cannot split them.

**Actual:** the image build and the CLI build resolve `git describe` independently. A tag pushed between them changes the answer, and the activation gate fails on every PR that contains the tagged commit until the whole workflow is re-run.

## Reproduction Steps

1. Push a release tag (for example `v0.0.121` at `673f815e0`).
2. Open or update a PR whose head contains that tagged commit, for example by merging `main`.
3. Let the `Publish complete managed images` workflow run.
4. `PR exact all-agent managed runtime activation` fails with the release mismatch above.
5. Re-run only the failed jobs -- it fails again, because the previously built images are reused.
6. Re-run the entire workflow -- it passes, because the images are rebuilt with the tag present.

Observed on run [34316774901](https://github.com/NVIDIA/NemoClaw/actions/runs/34316774901): attempts 1 and 2 failed, attempt 3 (full re-run) succeeded, with no change to the commit under test.

## Environment

- Runner: GitHub-hosted CI (`Publish complete managed images` workflow)
- Affected check: `PR exact all-agent managed runtime activation`
- NemoClaw commit under test: `624480df1b2c0a65e70152221bdd784ceb224214` (PR #11166)
- Release tag involved: `v0.0.121` at `673f815e0`
- Not host-specific; this is a CI/release-tooling ordering issue.

## Debug Output

Not applicable -- the failure happens in CI before any sandbox exists, so `nemoclaw debug` has nothing to collect. The relevant evidence is the workflow log quoted below.

## Logs

```text
2026-09-09T06:19:35Z * [new tag] v0.0.121 -> v0.0.121
2026-09-09T06:20:37Z [e2e target="managed-image-activation"] [phase 2/12] completed:
onboard and exercise OpenClaw — failed in 4s
2026-09-09T06:20:38Z Error: Sandbox workload preparation failed: managed image catalog contract
for 'openclaw' belongs to 'v0.0.120-83-g624480df1',
not 'v0.0.121-7-g624480df1'
```

Correlation with tag ancestry rather than with the change under test:

```text
PR #11166 head 624480df1 contains v0.0.121 tag commit -> activation failure
PR #11266 head 676221b91 behind that commit by 1 -> activation success
```

## Suggested Fix

Resolve the release once per workflow run and have every later step consume that value, instead of each step calling `git describe` on whatever tag state its checkout happens to have. Either of these would close the window:

- Fetch tags before the first step that stamps a release, and pin the resolved release into the job outputs / environment for the image build, the CLI build, and the activation check.
- Or fail fast: if the CLI build's resolved release differs from the release recorded on the candidate images, report that as a release-stamping inconsistency in CI rather than surfacing it as an onboarding preparation error in an E2E scenario, which makes it look like a product bug in the PR under review.

The second point matters for triage: as it stands this failure is indistinguishable, from the PR author's side, from a real regression in managed-image onboarding.

## Checklist

- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate

Contributor guide

Open the contributing guide

Research direction

Start with the `Publish complete managed images` workflow and the CLI build steps, then trace `getVersion()` and `git describe --tags --match "v*"`; `preparation.ts` shows where the release mismatch is rejected. Verify how tags are fetched and how values move between the image build, CLI build, and activation check. Done means one workflow run uses one resolved release for all three stages, or reports a clear release-stamping inconsistency.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, github-actions, typescript
Domain
ci-cd, release, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.