ci(e2e): keep exact-SHA base-image publication alive for its dependent run
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Problem
An exact-SHA E2E main run continues while competing base-image publications can cancel its required publication.
The workflow concurrency contracts conflict:
- `.github/workflows/e2e.yaml` does not cancel an in-progress main run.
- `.github/workflows/base-image.yaml` uses `base-image-${{ github.ref }}` and cancels an in-progress non-tag run.
A later merge can therefore turn an exact-SHA E2E run into a deterministic dependency failure even when its tests and image builds are passing. A rerun of an older publication can also cancel the publication for a newer SHA because every non-tag run shares the same group.
## Recent evidence
| E2E run | E2E SHA | Base-image run | Outcome |
| --- | --- | --- | --- |
| https://github.com/NVIDIA/NemoClaw/actions/runs/32190609485 | `96e05c7b69d5` | https://github.com/NVIDIA/NemoClaw/actions/runs/32178707193 | The applicable publication for `b2d1ce5` concluded `cancelled`; the gate failed before the dependent matrix ran. |
| https://github.com/NVIDIA/NemoClaw/actions/runs/32191102944 | `5f9269683a1d` | https://github.com/NVIDIA/NemoClaw/actions/runs/32191102997 | Twenty-one image jobs passed. A later push cancelled the final cohort-promotion job after 1h 20m 24s; the E2E gate then failed. |
After that cancellation, actor `ericksoa` started attempt 2 of the older `5f92696` publication. The older attempt re-entered the branch-wide concurrency group and cancelled https://github.com/NVIDIA/NemoClaw/actions/runs/32197181654 for newer SHA `826242285cf7` after 80 seconds.
For the second case, the dependency's final OpenClaw amd64 build and validation passed. The newer main commits landed 16 seconds apart immediately before final promotion.
This is a workflow/candidate-drift infrastructure failure, not a flaky test and not the failed-job rerun defect tracked by #9529.
## Required behavior
1. An in-progress exact-SHA E2E run must not lose its required base-image publication because another main push or rerun starts a competing publication.
2. The E2E run must continue to consume only base-image contracts bound to its exact source SHA and validated cohort.
3. A publication attempt must not overwrite, relabel, promote, or cancel work for a different SHA.
4. Keep dependency cancellation or missing evidence visible and fail closed.
5. Preserve the current E2E test selection and release judgment.
## Investigation
- Measure the queue and runner cost of allowing an in-progress publication to finish.
- Determine whether publication work can be skipped when a push does not change any base-image input.
- Compare exact-SHA concurrency groups, bounded serialization, and content-addressed publication reuse.
- Identify the atomic promotion boundary for a complete multi-platform, all-agent cohort.
- Quantify wasted runner time from cancelled publications and dependent E2E runs.
## Acceptance criteria
- Deterministic workflow tests model two main pushes while the first exact-SHA E2E run and base-image publication are active.
- A second test models a rerun of an older publication while a newer SHA's publication is active.
- The first E2E run retains a valid dependency for its own SHA after the second push.
- The second push cannot consume or overwrite the first SHA's contracts.
- The solution retains bounded concurrency and reports queue time and cancelled work.
- The change does not add a complete workflow rerun, broaden retries, add a live E2E test, or change release judgment.
Parent epic: #9159
Contributor guide
Assessment
This issue has not been assessed yet.