NVIDIA / NVIDIA/NemoClaw

ci(e2e): reconcile successful same-diff reruns with immutable PR gate checks

Open
#7,206 0 comments 0 reactions 1 assignee Claimed by @apurvvkumaria View on GitHub
area: ci area: e2e v0.0.127
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Summary

The trusted PR E2E path can verify a successful, exact-diff failed-job rerun while the PR remains permanently blocked by the immutable failure recorded before that rerun completed.

## Evidence

PR: #7015
Exact PR head: `e0100d4448c5537a53d8157df831cd67bf9e3e9d`
Exact base: `3c4ea5a1e3c37ae6688399b823258e1c31790fd2`

- The original required coordination check, [88213091950](https://github.com/NVIDIA/NemoClaw/runs/88213091950), completed with `failure` after the selected child run initially reported a failing `channels-stop-start (hermes)` lane.
- The selected child [E2E run 29696249841](https://github.com/NVIDIA/NemoClaw/actions/runs/29696249841) was retried once at the failed-job boundary. Attempt 2 passed the selected lanes and produced accepted evidence.
- The trusted [controller run 29696235538](https://github.com/NVIDIA/NemoClaw/actions/runs/29696235538) subsequently verified the same exact PR identity and completed successfully.
- The original required coordination check correctly remains immutable, but no fresh successful required check is created for the verified retry lineage. The PR therefore remains blocked even though the authorized retry passed.

This is adjacent to #7052/#7054, which added safe same-diff recovery for a bounded retry taxonomy, and #7140/#7146, which classify hosted-runner loss. The missing case is reconciliation after a GitHub failed-job rerun succeeds for the already-selected child run.

## Expected behavior

After a single authorized failed-job rerun succeeds, the trusted controller should create a fresh required check for the same immutable head/base identity only after it verifies the complete retry lineage and accepted evidence. The completed failure must remain unchanged for audit history.

## Acceptance criteria

- Never mutate or erase the original completed check.
- Recognize only one explicitly authorized failed-job rerun of the already-selected child run.
- Revalidate exact PR number, head SHA, base SHA, selected target plan, controller workflow identity, attempt lineage, and evidence before accepting the rerun.
- Require every selected lane and the final evidence verdict to pass on the accepted attempt.
- Create at most one fresh successful required check for that exact identity and retry lineage.
- Reject mixed identities, a different child run, additional reruns, missing evidence, ambiguous history, or partial success.
- Add adversarial tests for stale head/base, duplicate attempts, changed target plans, incomplete evidence, and replayed lineage.
- Document the recovery path alongside #7054's immutable-check behavior.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.