[Epic] Restore full E2E qualification on main
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
> [!IMPORTANT]
> **Want to claim work?** Open a child issue from the [Sub-issues view](https://github.com/NVIDIA/NemoClaw/issues/10979#sub-issues-issue-container), click its **Assignees** field or gear icon, then search for and select your own GitHub username. There may be no button literally named “Assign myself.”
## Outcome
Restore NemoClaw's full main E2E suite to a trustworthy release-qualification signal with zero unexplained primary failures.
Reference failure: [E2E full main run 33730208977](https://github.com/NVIDIA/NemoClaw/actions/runs/33730208977), candidate `3076188279c0ea657ec77aeeabebf1f82061eaaf`.
## Baseline
- Scorecard: 69 passed, 25 failed, 11 skipped; 94 of 105 targets executed.
- One failure is the downstream Release qualification aggregate, leaving 24 primary failed jobs.
- [Later main run 33761515943](https://github.com/NVIDIA/NemoClaw/actions/runs/33761515943) reproduces 20 of the primary failures with matching signatures.
- Those 20 jobs originally reduced to 13 deterministic root-cause trackers.
- One additional environment/configuration failure and three transient reliability failures also required explicit disposition.
## Exact-candidate recalibration
The [PR #11071 E2E run](https://github.com/NVIDIA/NemoClaw/actions/runs/33915961523) tested exact candidate `0ddc3ea52efe8a5d63099b92f43cbe0a43c88a01`. It completed with 64 successful jobs, 28 failed jobs, and 16 skipped jobs. All 28 were actual failed jobs; the protected managed-image GPU and local-inference job was an independent build failure, not an aggregate.
### Cleared or not reproduced
- #10963 is fixed and closed by merged #11071. Hermes E2E passed, and the other original Hermes targets moved past `SECRET_BOUNDARY_REFUSED`.
- #10965 did not reproduce: both Hermes rebuild targets passed. Keep it open until the pass is reconciled with main history and confirmed on current main.
- #10974 has one passing Sandbox operations target after #10695 merged. Repeated main evidence is still required.
- #10975 is fixed by merged #11022, but the exact-staging jobs were skipped. Final staging evidence remains outstanding.
### High-fanout current blockers
- #11084 blocks six initial onboarding targets because new dashboard or Hermes API host forwards refuse connections.
- #11074 blocks four reuse, resume, and token-rotation targets because the expected existing dashboard listener is treated as a foreign port conflict.
- #10964 and #10970 still reproduce unchanged across five upgrade targets. Reduced PR #10996 owns these fixes.
- #10971 blocks both OpenClaw and Hermes channel lifecycle during initial Google Chat provider registration.
### Single-target and layered blockers
- #10966 was not reverified. Pi AMD64 failed earlier at #11083's conflicting catalog authorities, and Pi Arm64 did not execute.
- #10967 and #10968 both reproduce in Messaging providers; #11011 and #11000 own those fixes.
- #10969 reproduces the retired-model HTTP 410; #11070 supersedes closed #10918.
- #10681 reproduces in Dashboard remote bind with the exact `GATEWAY_UNSAFE_CONFIG_PATH` failure.
- #10972, #10973, #11081, #11082, #11085, and #11086 each have one exact failed target.
- #10978 was not reverified. Hosted inference failed earlier at #11084's dashboard-forward check.
- #11088 tracks the separate protected OpenClaw image build that exhausted its npm-audit completeness retry.
## Deterministic release blockers
- [x] #10963 — Hermes managed restart rejects required path variables — fixed by merged [#11071](https://github.com/NVIDIA/NemoClaw/pull/11071) (@prekshivyas)
- [ ] #10964 — Legacy OpenClaw upgrades are rejected as DCode prepared contexts — candidate [#10996](https://github.com/NVIDIA/NemoClaw/pull/10996) (@ericksoa)
- [ ] #10965 — Hermes rebuild journaling requires a nonexistent onboarding session — not reproduced; current-main confirmation needed
- [ ] #10966 — Pi qualification receipts are not anchored to the exact candidate — blocked before assertion by #11083
- [ ] #10967 — Messaging accepted-extras breadcrumb is not emitted — candidate [#11011](https://github.com/NVIDIA/NemoClaw/pull/11011) (@apurvvkumaria)
- [ ] #10968 — Slack proof workspace omits hoisted `fast-uri` — candidate [#11000](https://github.com/NVIDIA/NemoClaw/pull/11000) (@AzeelSajjad)
- [ ] #10969 — Model Router selects a retired Nemotron model — candidate [#11070](https://github.com/NVIDIA/NemoClaw/pull/11070) (@rsliter)
- [ ] #10681 — OpenClaw startup recovery rejects the config path after restart — reproduced; no focused candidate
- [ ] #10970 — v0.0.115 Shields sandbox cannot complete receipt migration — candidate [#10996](https://github.com/NVIDIA/NemoClaw/pull/10996) (@ericksoa)
- [ ] #10971 — Google Chat profile and credential boundary disagree for OpenClaw and Hermes
- [ ] #10972 — EXDEV plugin fixture is unreadable — candidate [#10808](https://github.com/NVIDIA/NemoClaw/pull/10808) (@rsliter)
- [ ] #10973 — Ollama recovery expects a breadcrumb product does not emit — candidate [#11013](https://github.com/NVIDIA/NemoClaw/pull/11013) (@apurvvkumaria)
- [ ] #10974 — Sandbox credential-boundary scan exceeds its fixed timeout — patch merged in [#10695](https://github.com/NVIDIA/NemoClaw/pull/10695) (@ericksoa); one exact-candidate pass
- [ ] #11074 — Re-onboarding rejects its expected existing ForwardTcp dashboard listener — four exact failed jobs
- [ ] #11081 — Hermes MCP restart cannot reuse its stored provider credential
- [ ] #11083 — Pi fixture supplies conflicting managed-image catalog authorities — candidate change in #10996
- [ ] #11084 — New host forwards refuse connections during deployment verification — six exact failed jobs
- [ ] #11085 — Repaired OpenClaw sandbox loses canonical CLI pairing
- [ ] #11086 — Network-policy denial no longer exposes the asserted HTTP 403
## Environment qualification
- [x] #10975 — Exact-staging Brev target receives an invalid NVIDIA credential shape — resolved by merged [#11022](https://github.com/NVIDIA/NemoClaw/pull/11022) (@jyaunches); rerun still required because staging was skipped
## Reliability and failure classification
- [ ] #10976 — Telegram sandbox creation Ready/durable-ID race
- [ ] #10977 — Gateway guard-chain recovery health-timeout flake
- [ ] #10978 — Hosted-inference outage fails OpenClaw posture qualification — candidate [#11077](https://github.com/NVIDIA/NemoClaw/pull/11077) (@cjagwani); exact run blocked earlier by #11084
- [ ] #11082 — Hermes security child-process census changes after restart
- [ ] #11088 — Protected OpenClaw image build exhausts npm-audit completeness retries
## Claim a child issue
GitHub does not show a literal **Assign myself** button in every issue layout.
1. Open the [native Sub-issues view](https://github.com/NVIDIA/NemoClaw/issues/10979#sub-issues-issue-container).
2. Open an unassigned child issue.
3. Click **Assignees** or the gear icon beside it in the issue metadata sidebar.
4. Search for your own GitHub username and select it.
5. Comment with the intended scope and link the fixing PR when it is opened.
CLI fallback:
```bash
gh issue edit ISSUE_NUMBER --repo NVIDIA/NemoClaw --add-assignee @me
```
Claim the specific child issue rather than assigning yourself to this epic.
## Recommended execution order
1. Land the high-fanout forward fixes in #11084 and #11074, then rerun the targets they mask.
2. Reduce and validate #10996 for #10964 and #10970. Keep #11083 there only if its focused evidence stays small; otherwise split it.
3. Rerun Pi after #11083 so #10966's exact receipt contract is actually exercised on both architectures.
4. Resolve the remaining deterministic single-target blockers and validate each affected target.
5. Rerun exact staging to complete #10975 environment qualification.
6. Reclassify #10976, #10977, and #10978 using runs that reach their intended assertions.
7. Run the complete exact-head suite without narrowing or skipping affected coverage.
## Guardrails
- Do not close a child only because its E2E target was skipped, masked by an earlier failure, or removed.
- A test-contract fix must retain the user-visible, security, migration, or lifecycle behavior the target protects.
- External/provider failures must be classified separately without weakening fail-closed product assertions.
- Every child must link its fixing PR and targeted validation evidence before closure.
## Definition of Done
- [ ] All deterministic child issues are closed with merged fixes and passing target evidence.
- [ ] The Brev exact-staging target passes using the intended environment credential.
- [ ] Each transient issue has either a confirmed product fix or a reviewed CI resilience/classification fix.
- [ ] Two consecutive full main E2E runs complete with zero primary failed targets.
- [ ] Release qualification is green.
- [ ] Final scorecards and run links are posted to this epic.
## Ownership
Candidate owners are listed beside linked PRs. Items without a candidate remain unassigned pending triage.
Contributor guide
Research direction
Start with the referenced full main E2E runs 33730208977 and 33761515943, then review the Sub-issues view for a specific unassigned child issue. Work should be scoped to one child blocker, with its fixing PR and targeted validation evidence linked. The epic is done after two consecutive full main runs have zero primary failures and release qualification is green.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, typescript
- Domain
- ci-cd, devops, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100