Unify executable E2E inventory and retire redundant test infrastructure
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Problem and desired outcome
NemoClaw contributors must understand several E2E registration and selection systems before adding or changing a test. Declared targets can become empty skipped tests. Workflow validators duplicate execution details, and compatibility code outlives its original callers.
Create one discoverable inventory of executable E2E targets. Each target identifies its behavior, selection metadata, and execution route. Vitest continues to own test execution and fixtures. GitHub Actions continues to own jobs, permissions, credentials, and dependencies between jobs.
A contributor should be able to find a test's execution route and add a supported variant without choosing another registration system. A newly discovered E2E entry point with no disposition must fail the registration check. An explicitly selected unknown or removed target must fail with an actionable error.
This issue replaces #8281 and its remaining open E2E children: #8285, #8286, #8287, and #8288. Closing those issues records supersession; it does not certify their implementation as complete. The five already-closed children of #8281 remain historical records.
## Baseline and limits of the audit
Baseline: `main` commit [`48cefc6f53bc158fae59ee319718fc00345f660a`](https://github.com/NVIDIA/NemoClaw/commit/48cefc6f53bc158fae59ee319718fc00345f660a), inspected on 2026-09-10. Refresh this baseline before implementation.
| Observation | Current evidence |
| --- | --- |
| Live test files | 86 files under `test/e2e/live/`; filesystem and Vitest file discovery agree. |
| Homogeneous catalogue | 61 target entries referencing 54 test files. |
| Typed registry | 26 declarations; four are wired into the live driver, and 22 become empty skipped tests. |
| Tagged shared tests | Two integration files: buildless managed-image onboarding and vLLM Docker storage. |
| Workflow validation | 20 `tools/e2e/*workflow-boundary*.mts` files contain 11,040 lines. The 33 matching test files contain 8,550 lines. These filename-based counts are an audit baseline, not a deletion allowance. |
| Main execution workflows | `e2e.yaml` has 6,373 lines; `e2e-standard-profile.yaml` has 738 lines. |
The four executable typed targets are:
- `ubuntu-policy-custom-missing-presets-negative`
- `ubuntu-repo-cloud-langchain-deepagents-code`
- `ubuntu-repo-cloud-openclaw`
- `ubuntu-repo-docker-post-reboot-recovery`
The planner currently combines catalogue entries, typed-registry scenarios, tagged shared tests, and metadata discovered from dedicated workflow jobs. The advisor also consumes separate sources. See the [planner](https://github.com/NVIDIA/NemoClaw/blob/48cefc6f53bc158fae59ee319718fc00345f660a/tools/e2e/workflow-plan.mts), [catalogue](https://github.com/NVIDIA/NemoClaw/blob/48cefc6f53bc158fae59ee319718fc00345f660a/tools/e2e/target-catalogue.mts), and [registry support checks](https://github.com/NVIDIA/NemoClaw/blob/48cefc6f53bc158fae59ee319718fc00345f660a/test/e2e/registry/runtime-support.ts).
The 86 files are not the complete execution inventory. Reconciliation must include:
- Separate image-publication, staging Launchable, portable-profile, Podman, platform, and qualification workflows.
- Producer jobs and external dispatch, including native-runtime qualification and Jetson.
- Script entry points, imported test modules, generated scenarios, and `test/e2e-runtime/` packaged-image tests.
- Manual hardware qualifications, including DGX Spark and experimental Windows tests.
- Conditional collection, selectors, skips, exclusions, and compatibility aliases.
The preliminary file audit traced 20 files outside the catalogue, typed entry point, and marked main-workflow jobs. Some run elsewhere; others are manual, excluded, or compatibility code. File discovery and references do not prove successful execution or complete case selection. Case-level and workflow/script reconciliation remain required.
## Decisions and scope
### One executable inventory
Use one target model for identity, observable behavior, applicable agent/environment/runtime variants, changed-file ownership, scheduling eligibility, and execution route. Keep definitions near their owners when useful; physical colocation in one large file is not required.
An execution field can select an existing reusable profile or a specialized workflow. Generate workflow matrices, manual selectors, advisor recommendations, and execution summaries from that model. A workflow with additional credentials or dependencies remains discoverable through the same inventory.
Preserve rich onboarding scenario data where the live driver consumes it. Preserve automatic discovery where it supplies reliable metadata. The final design must remove the superseded registration and selection paths; wrapping all four inventories behind another function is insufficient.
Do not add another test runner, workflow compiler, generic execution language, persistent migration ledger, or new registry alongside the old ones. The migration disposition report is review evidence generated from a pinned revision.
### Resolve declarations and leftovers
For each existing entry, record its behavior, current caller, execution or replacement, proposed action, and supporting evidence:
| Existing item | Required disposition |
| --- | --- |
| Executable automated or manual test | Retain its execution route and prerequisites. Preserve current scheduling unless a separate decision changes it. |
| Unimplemented declaration | Find any existing coverage of the same behavior. Preserve an accepted missing requirement in an owned issue, then remove the empty declaration and unused data. A placeholder alone does not establish a requirement. |
| Compatibility selector or entry point | Inspect current callers and pinned trusted controllers. Record the supported cutoff. Remove the shim after callers are retired; retain any unique assertion at its current behavior owner. |
| Obsolete or superseded test | Prove the behavior was removed or identify equivalent replacement coverage before deletion. Compare assertions and environment, not names alone. |
| Unresolved behavior or support decision | Record a concrete maintainer decision and owner before changing execution, support, or compatibility. Preserve existing executable evidence meanwhile. |
Remove empty skipped-test registration for unsupported declarations. Manual tests remain executable targets with explicit requirements. Missing fixtures and proposed product combinations belong in planning records, not the executable catalogue. Ordinary environment-dependent skips must remain distinguishable from successful coverage.
### Delete redundant validation with its execution duplication
Carry forward the [maintainer correction on #8288](https://github.com/NVIDIA/NemoClaw/issues/8288#issuecomment-5269168159): table-driving YAML validators preserves the wrong abstraction.
Group executions with the same operational requirements under existing reusable profiles. Add another profile only for a current consumer that cannot use an existing one. Delete superseded job YAML, validator clauses, and tests that merely assert their literal structure in the same change.
Retain a small policy check for pinned actions, minimum permissions, trusted secret boundaries, and artifact ownership. Test executable helpers, security denials, and product evidence at their enforcing boundaries. Artifact authenticity and runtime-identity validation still have real consumers and must remain.
GitHub Actions owns execution dependencies and job status. Select required executions from the canonical model. Verify required receipts and outcomes without reconstructing another inventory from workflow text.
### Carry forward the remaining cleanup from the old issues
- **#8285:** Re-audit before deletion. The old `runtimeCase`, `compileRuntimeMatrix`, `executeRuntimeCaseThroughProvider`, and `writeExecutionEvidence` symbols are absent at this baseline. Today's `fixtures/runtime-provider.ts` and `support/runtime-matrix.test.ts` have real Docker/Podman consumers. Preserve current runtime behavior, compatibility checks, and generic artifact redaction. Remove only proven unused remnants.
- **#8286:** Remove the 22 unsupported declarations and their unused manifests, expected states, dimensions, and empty tests. Preserve the four executable contracts. Verify each deletion against accepted scope and alternative live coverage.
- **#8287:** Retire old selector/controller paths after verifying their cutoff. Separately migrate the packaged-image snapshot probe to the current migration-state/snapshot-management behavior before deleting the legacy module. `test/e2e-runtime/managed-image-openclaw-security.test.ts` still imports `/opt/nemoclaw/dist/blueprint/snapshot.js`; deletion must preserve rollback and security evidence.
- **#8288:** Replace `test/e2e/mock-parity.json` with discovery/colocated defaults and explicit exceptions, preserving detection of changed live helpers without a fast-test disposition. Simplify workflow validation as described above. Share fake HTTP/WebSocket mechanics only where current consumers have the same contract; keep protocol payloads, state, assertions, and security behavior with their owners.
## Delivery slices
Each slice must leave a usable result and include its tests, deletion, and owning guidance. Do not merge an unused abstraction awaiting later migration.
1. **Make registry output truthful.** Refresh the disposition audit, resolve the placeholder decisions, preserve the four runnable contracts, and remove empty declarations and unused remnants. Verify the runnable matrix before and after.
2. **Make every executable target discoverable through one model.** Migrate registration, selection, and reporting by complete groups. Include specialized and manual execution routes. Add independent completeness checks as groups migrate. Remove old discovery/selection ownership for each migrated group.
3. **Reduce execution and validation duplication.** Consolidate compatible jobs into reusable profiles, deleting their redundant YAML and validators together. Preserve specialized credential and multi-job boundaries with direct evidence.
4. **Complete compatibility retirement.** Apply recorded selector cutoffs and migrate the live snapshot consumer. Preserve rollback and security behavior before deleting obsolete modules and entry points. Selector and snapshot work may land separately.
5. **Simplify support-test maintenance.** Replace the mock-parity ledger and consolidate proven transport duplication in separate, independently useful changes. Delete the superseded mechanisms in each change.
The audit is the first deliverable of slice 1. It is not a new runtime component. Slices 4 and 5 can proceed independently where their decisions and consumers are established. Record net maintained source-and-test changes for each PR; cleanup should reduce the total. Any growth must identify its necessary consumer and protecting contract. Historical LOC savings estimates are superseded by the refreshed audit.
## Acceptance and validation
- [ ] A pinned baseline accounts for every discovered live module, generated case/variant, external E2E entry point, and current execution route.
- [ ] Every old item has a retained execution, proven replacement, approved retirement, or an owned unresolved decision. No item disappears because one of the four old inventories omitted it.
- [ ] Independent filesystem/Vitest discovery agrees with registration. Workflow/script consumers are reconciled independently. Unknown registrations, dangling references, and unexplained unregistered entries fail.
- [ ] Parameterized cases, imported modules, selectors, and environment-gated collection are checked. Registering a file does not substitute for selecting its intended tests.
- [ ] Before/after plans preserve default, changed-file, explicit manual, Docker, Podman, agent, and architecture coverage, except for reviewed dispositions.
- [ ] One model supplies execution selection, advisor recommendations, and coverage reporting. Superseded registration paths and empty placeholder tests are removed.
- [ ] Explicit unknown/removed selectors fail with a specific error. An empty selection or all-skipped selection cannot certify a required execution.
- [ ] Required artifact and execution failures remain failures. Unavailable infrastructure, inconclusive verification, product failure, and successful evidence remain distinguishable.
- [ ] Retained security tests prove denied credentials, unsafe inputs, wrong artifact identity, forbidden permissions, and unsafe cleanup remain denied at the enforcing boundary.
- [ ] Manual qualifications remain discoverable without automatically enabling unsupported platforms, credentials, or billable resources.
- [ ] Compatibility deletion includes evidence about pinned controller callers. Snapshot replacement includes packaged-artifact rollback evidence.
- [ ] Repository guidance, skills, and E2E READMEs describe the resulting ownership. Public documentation follows the repository documentation workflow.
- [ ] The completed work has a measured net reduction in maintained infrastructure and tests without losing unique behavior or security coverage.
Use the narrowest stable checks: E2E-support, planner/advisor tests, project membership, repository checks, applicable CLI/plugin type checks, and packaged-image tests for snapshot changes. Compare generated plans across explicit representative inputs. Use collection in an isolated environment where importing a module can touch live prerequisites.
Run live E2E when a slice changes a real execution, provisioning, credential, cleanup, process, image, or GitHub Actions boundary. Bind those results to the tested revision. Do not require a full live run for metadata-only equivalence that deterministic checks can establish.
## Security, compatibility, and ownership
This is core contributor/maintainer infrastructure maintenance requested by @cv. The implementation owner is unassigned. NemoClaw maintainers own the resulting inventory and workflows; each retained target keeps an accountable behavior owner.
This issue authorizes no new supported platform, runtime, agent integration, or credential boundary. Product-scope changes and compatibility cutoffs require recorded maintainer decisions. Experimental/manual qualifications do not establish product support.
Preserve trusted workflow versus candidate-code separation. Workflow secrets must stay in their approved jobs/steps and existing credential stores, with the existing lifetime and cleanup. Keep credentials out of candidate-controlled planning, logs, reports, and artifacts. Preserve pinned dependencies and artifact identity, restrictive defaults, TLS verification, redaction, and required status propagation.
For execution or cleanup failures, preserve ownership checks and failure classification. Reconcile ambiguous remote state before a retry or destructive action. This consolidation adds no retry policy and widens no cleanup authority.
## Related work and retirement
- Supersedes #8281, #8285, #8286, #8287, and #8288. Preserve their discussion as historical rationale.
- The completed/closed #8281 children (#8282, #8283, #8291, #8292, #8345) are not reopened by this issue.
- #7912 retains performance budgets, runner experiments, and scheduling-policy changes. This issue unifies the metadata and execution ownership those policies consume.
- #7614 and #7451 are closed historical work. Preserve their retained-versus-removed coverage decisions and completed feedback-loop changes; this issue does not reopen them.
- #7744 owns native-runtime product qualification. Preserve its current trusted controller, collector, matrix, and activation evidence. The historical overlap has changed: #8062 is merged and #8079 is closed; they are not pending-PR blockers to copy into this plan.
- #11397 is the separate OpenShell SDK setup fix currently touching the reusable catalogue workflow. Reconcile its final result before changing that workflow; this issue does not replace the fix.
Close the superseded issues with a link here and state that remaining delivery moved. Do not mark the cleanup implemented merely because its old tracking issues were retired.
Contributor guide
Research direction
Refresh the pinned baseline audit, then read tools/e2e/workflow-plan.mts, tools/e2e/target-catalogue.mts, and test/e2e/registry/runtime-support.ts. Reconcile workflow, script, generated, manual, and packaged-image entry points before changing ownership. Done means one complete executable inventory, truthful registry output, preserved runnable contracts, and removed redundant selectors and validation with supporting tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, typescript
- Domain
- ci-cd, testing, tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100