Retire or consolidate six redundant full-matrix E2E jobs
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Problem
Full-matrix main run [35118412060](https://github.com/NVIDIA/NemoClaw/actions/runs/35118412060) reported 29 primary failed jobs. Six of those jobs no longer justify an independent full-matrix execution slot after recent product-ownership decisions and coverage consolidation.
Failures are not themselves a reason to delete coverage. Each job below is included because its product contract has moved to a native owner, is already proved by another retained live boundary, or is deterministic and belongs in a smaller test lane.
## Product Decision and Outcome
**Decision: Accept.**
- **Reason:** remove six redundant or obsolete full-matrix jobs while preserving every supported observable, recovery, cleanup, and security outcome at its current authoritative boundary.
- **Placement:** NemoClaw E2E workflow, target catalogue, live fixtures, deterministic tests, and owning E2E documentation.
- **Accountable owner:** NemoClaw E2E maintainers, coordinated with the owners of #11766 and #11905.
- **Validation:** exact-head deterministic owners, targeted retained live lanes, and one completed full manual main-equivalent run must prove the reduced matrix before the final deletion merges.
## Jobs and Required Dispositions
| Current job or target | Disposition | Required retained owner | Existing authority or overlap |
| --- | --- | --- | --- |
| `openclaw-plugin-runtime-exdev` | Delete the dedicated job, target, live fixture, and support contracts. | `full-e2e`, `rebuild-openclaw`, `rebuild-hermes`, managed-image activation, and messaging-provider native lifecycle proofs. | Accepted native plugin/package ownership in #11766; implementation in #11906. |
| `ubuntu-repo-docker-post-reboot-recovery` | Delete the current direct-Docker simulated-reboot target. Preserve only supported state survival through the retained OpenShell lifecycle owner. | `openshell-stop-start-lifecycle` plus deterministic exact-identity, cleanup, and state-preservation tests. | Accepted OpenShell lifecycle ownership in #11905; implementation coordination with #11909. |
| `issue-2478-crash-loop-recovery` | Delete the dedicated NemoClaw recovery job. If native self-recovery remains a supported qualification outcome, retain one bounded observation in an agent-owned qualification lane without NemoClaw repair authority. | `full-e2e` or the applicable native-agent qualification owner, plus deterministic NemoClaw no-intervention coverage. | Native agents own gateway and background-process lifecycle under #11905 and its prerequisite ownership decisions. |
| `Shared E2E (onboard-managed-image-buildless-e2e-docker)` | Remove this deterministic integration test from full live-matrix dispatch. Do not delete its useful deterministic coverage. | Ordinary CI execution of `test/onboarding/onboard-managed-image-buildless-e2e.test.ts`. | The failed assertion was stale documentation prose and exercised no real sandbox, network, process, or service boundary. |
| `device-auth-health` | Delete the standalone live job after moving or confirming all 401/status classification behavior at deterministic owners. Retain one real dashboard observation only if it is not already covered. | `src/lib/verify-deployment.test.ts`, gateway HTTP readiness/reuse tests, status health tests, and `dashboard-remote-bind` for the remaining live dashboard boundary. | Existing deterministic 401 coverage and overlapping dashboard plus gateway-recovery E2Es. |
| `cloud-inference` | Delete the standalone job after disposing of its unique assertions. | `full-e2e` and `bootstrap-install-smoke` for install/onboard/inference; security tests for credential isolation; deterministic image/skill contracts for filesystem shape. | The live install, PATH, onboard, and hosted-inference outcomes already have retained live owners. Repo skill validation and exact filesystem shape do not require a live provider. |
## Scope
### Included
- Remove the six independent workflow jobs or dispatch entries listed above.
- Delete job-specific live fixtures and support contracts that have no remaining consumer.
- Move deterministic behavior to the smallest existing unit, integration, package-contract, or `e2e-support` owner before deleting live assertions.
- Consolidate unique live behavior into an existing retained target only when it represents a distinct supported external boundary.
- Update the workflow planner, target catalogue, mock-parity data, assertion census, selector validation, documentation, and report aggregation.
- Record before-and-after target, job, live-test, assertion, runtime, and line counts.
### Excluded
- Do not retire either historical upgrade target from run 35118412060. The v0.0.115 Shields retirement migration was intentionally retained after Shields left core, and the v0.0.89 target remains an old-to-current compatibility boundary.
- Do not retire snapshot/restore. Broader snapshot removal is explicitly outside #11905.
- Do not retire `openshell-stop-start-lifecycle`; it is required validation for #11905.
- Do not retire OpenClaw or Hermes security-posture qualification.
- Do not remove MCP, Pi, GPU Ollama, Model Router, Brave Search, provider switching, staging Launchable, performance, CLI routing, bootstrap installation, dashboard remote-bind, or protected managed-image GPU coverage under this issue.
- Do not weaken fail-closed cleanup, exact ownership, secret redaction, negative policy, or credential-isolation evidence.
- Do not add replacement jobs, registries, compatibility layers, broad retries, or new matrix permutations.
## Coverage-Transfer Requirements
For every removed live assertion, record exactly one disposition:
1. already covered by a named retained deterministic or live test;
2. moved to a named smaller test in the same change;
3. covered by another retained observable behavior test; or
4. no quality value because it checks incidental output, internal representation, or third-party behavior.
Do not move assertions into helpers, snapshots, aggregate receipts, shell probes, or opaque composite checks merely to reduce the visible count.
## Acceptance Criteria
- [ ] The six listed jobs no longer execute in a full manual E2E run.
- [ ] `openclaw-plugin-runtime-exdev` removal is delivered through or reconciled with #11906 and closes or updates the conflicting retention record in #11547.
- [ ] Post-reboot and crash-loop removal is delivered through or reconciled with the final E2E inventory required by #11905 and #11909.
- [ ] The buildless onboarding test remains in ordinary deterministic CI and is no longer reported as a live full-matrix E2E job.
- [ ] HTTP 401 means reachable at every supported status/readiness boundary, proved by focused deterministic tests.
- [ ] Dashboard remote-bind retains its distinct real dashboard exposure and audit outcome without inheriting unrelated recovery assertions.
- [ ] Full E2E or another retained live owner proves install, onboarding, sandbox inference, and the relevant credential boundary formerly repeated by `cloud-inference`.
- [ ] Every deleted assertion has a reviewed coverage disposition and every moved behavior has fail-before/pass-after evidence at its new owner.
- [ ] Negative, recovery, cleanup, exact-owner, redaction, and security outcomes remain covered at their authoritative boundaries.
- [ ] Manual selectors for deleted targets are either rejected with the existing unknown-selector contract or handled by an explicitly time-bounded compatibility rule. No silent alias is added.
- [ ] Workflow planner, target catalogue, mock parity, semantic phase checks, assertion census, documentation, and report aggregation pass with no stale references.
- [ ] The completed change is net-negative in workflow, test, and support lines and introduces no replacement lifecycle or test infrastructure.
- [ ] Targeted retained live lanes pass on the exact final commit.
- [ ] One completed full manual E2E run on the exact final commit confirms the reduced matrix and reports no missing aggregate dependency.
## Security and Failure Boundaries
- Credential locations, access, lifetime, and removal do not change under this issue.
- Credential-isolation and redaction assertions removed from `cloud-inference` must have named retained owners before deletion.
- A failed or ambiguous cleanup remains a failure. Job consolidation must not replace exact identity and absence checks with name-only or best-effort cleanup.
- Device-authenticated HTTP 401 responses remain classified as reachable without exposing authentication material.
- Native gateway failure must not trigger reintroduction of a NemoClaw-owned process or container recovery controller.
- Workflow and external-resource writes remain restricted to existing trusted E2E boundaries. This issue adds no new credentials, permissions, or external resources.
## Delivery Order and Dependencies
1. Land or reconcile #11906, including replacement native plugin lifecycle evidence and #11547 disposition.
2. Complete the lifecycle-owned inventory in #11909 for the post-reboot and crash-loop jobs.
3. Relocate deterministic buildless and device-auth coverage.
4. Consolidate the remaining unique `cloud-inference` evidence into named retained owners.
5. Remove stale selectors, workflow aggregation edges, fixtures, and documentation together with the final jobs.
6. Run targeted evidence and one exact-head full manual E2E.
Stop rather than delete a job if discovery finds a unique supported external behavior without a retained owner, if #11906 or #11909 changes its accepted ownership boundary, or if the exact-head full run cannot prove the reduced aggregate.
## Related Work
- #11766 and #11906: native plugin and package lifecycle ownership; explicit EXDEV job retirement.
- #11905 and #11909: retirement of NemoClaw-owned Docker and Podman lifecycle orchestration.
- #11547: prior decision to retain the EXDEV live boundary; must be reconciled with the newer accepted native-ownership decision.
- #10517: completed high-cost E2E pruning, including intentional retention of the Shields retirement-upgrade boundary.
- #10920: assertion pushdown rules. It does not itself authorize target or job deletion.
- #10722: Shields product removal and retained legacy-state migration boundary.
- #11775: OpenShell capability-enforcement ownership with intentionally retained OpenClaw and Hermes security-posture E2Es.
## Checklist
- [x] The requested deletion set is bounded to six named jobs.
- [x] Each proposed deletion names a retained owner or a required ownership decision.
- [x] Conflicting and dependent issues and PRs are identified.
- [x] Security, cleanup, recovery, and credential evidence are preserved.
- [ ] Assign one implementation owner after overlapping PRs establish their final scope.
Contributor guide
Assessment
This issue has not been assessed yet.