NVIDIA / NVIDIA/NemoClaw

[Epic] Halve live E2E assertion bloat without weakening coverage

Open
#10,920 1 comment 0 reactions 0 assignees View on GitHub
area: architecture area: e2e
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Problem

NemoClaw's live E2E suite contains far more assertions than are needed to prove its observable product behavior. Many live assertions validate intermediate commands, exact output, internal state, process topology, registry fields, receipts, hashes, or other deterministic details that can be tested earlier and more reliably in the pipeline.

At `fe18b2f004e2304c52db33cf7bb5895828197f9a`, the `e2e-live` project collects 95 test cases and contains:

- 2,184 direct Vitest `expect(...)` calls in `test/e2e/live/**/*.test.ts`;
- an average of 22.99 direct assertions per collected E2E test;
- 2,677 `expect(...)` calls across live test and companion modules, before counting custom `assert*`, `throw`/`fail`, or generated shell, Python, and JavaScript probe conditions.

This assertion volume makes expensive E2E tests brittle and makes a failure more likely to identify an implementation checkpoint than a broken NemoClaw outcome.

## Goal

Reduce the direct live E2E assertion surface by at least 50 percent while preserving or improving NemoClaw's quality posture.

The closing targets for the same 95-test baseline are:

- no more than **1,092 direct assertions** in live test files;
- no more than **11.5 direct assertions per collected E2E test** on average;
- no more than **1,338 assertions** across live test and companion modules.

If independently authorized work changes the collected test count during this epic, report both the fixed 95-test baseline comparison and the normalized assertion average. Work under this epic must not remove E2E test cases, targets, or jobs merely to satisfy the numeric target.

The numeric target is not permission to combine many checks into one opaque assertion. The result must be fewer independently failing conditions, not different syntax around the same conditions.

## Quality-preservation rule

Every assertion removed from a live E2E must receive exactly one disposition in its pull request:

1. **Already covered earlier** — link the existing unit, integration, package-contract, workflow-boundary, or `e2e-support` test that detects the same failure earlier in the pipeline.
2. **Moved earlier** — add or strengthen a focused lower-level test in the same pull request, prove that it fails for the behavior formerly protected by the E2E assertion, and then remove the live assertion.
3. **Covered by another retained behavior test** — link the exact retained test and explain why the removed assertion provides no distinct regression signal.
4. **No quality value** — explain why the assertion checks incidental output, duplicated setup success, a third-party implementation detail, or an internal representation that is not a NemoClaw contract.

An assertion cannot be removed merely because it is brittle, slow, inconvenient, or difficult to maintain. If it protects unique supported behavior and no earlier test can detect that behavior, it remains live.

When an assertion moves earlier in the pipeline, its new owner must test the behavior rather than copy a literal source shape or mirror the implementation. Existing source-shape exception rules continue to apply.

## Assertions that normally remain live

Retain live assertions when they require the real installer, process, sandbox, runtime, hardware, network, external service, or lifecycle boundary and prove an observable outcome such as:

- onboarding completes and the selected agent can perform a turn;
- a provider or policy change affects real agent behavior;
- state remains usable after restart, rebuild, upgrade, or recovery;
- an allowed operation succeeds or a denied operation remains denied;
- one sandbox remains usable while another sandbox changes;
- credential or secret material is not observable across the protected boundary;
- cleanup removes the user-visible resource owned by the journey.

Preserve negative, failure, recovery, cleanup, and security outcomes when they are distinct live behaviors. Reduce checks of the exact mechanism enforcing those outcomes.

## Assertions that normally move earlier or are removed

Unless the detail is itself a reviewed public or security contract, live E2E should not assert:

- every intermediate command exit code after the harness has already failed the operation;
- exact progress, log, warning, or status strings;
- exact SHAs, digests, artifact names, provenance, or receipt schemas;
- PID, PPID, process owner, process topology, or replacement identity;
- internal registry, session, plan, route, provider, or configuration fields;
- exact container IDs, labels, image layers, tags, or temporary paths;
- exact file paths, hashes, ownership, or permission modes when behavior can prove the security property;
- parser, classifier, schema, validation, or serialization behavior;
- repeated observations of the same outcome through several internal surfaces;
- third-party implementation details that NemoClaw does not own.

## No assertion laundering

The following do not count as assertion reduction:

- moving `expect` calls into a helper or subprocess;
- replacing several assertions with a large `toMatchObject`, snapshot, or receipt comparison;
- combining several independently meaningful conditions into one boolean;
- converting matchers into `assert*`, `throw`, `fail`, shell `test`, or generated-probe conditions;
- returning a large internal-state object and asserting one aggregate verdict;
- deleting diagnostics required to understand a retained behavioral failure.

An assertion point is one independently failing condition regardless of language, wrapper, or process.

## Scope

This epic includes:

- assertions in `test/e2e/live/**/*.test.ts`;
- product assertions in live companion modules and generated probes;
- moving deterministic checks to existing or new lower-level tests;
- deleting assertions that have no distinct quality value;
- adding an assertion census and no-growth ratchet;
- recording before-and-after assertion counts and dispositions in every remediation pull request.

## Non-goals

This epic does not:

- split, combine, rename, retire, or reschedule E2E targets or jobs;
- change the E2E matrix, selectors, or execution tiers;
- optimize runner cost, wall time, sharding, retries, or infrastructure except as an incidental result;
- add new E2E scenarios or permutations;
- change product behavior;
- weaken a supported observable outcome or security property;
- duplicate the broader reliability work in #9159 or runtime and scheduling work in #7912.

## Proposed child work

- [x] Add a transitive live assertion census and no-growth ratchet.
- [ ] Complete and tighten the high-cost pruning already tracked by #10517 against this epic's quality-preservation rule and 50-percent target.
- [ ] Reduce MCP bridge and credential-lifecycle assertions to observable bridge behavior.
- [ ] Reduce rebuild assertions to state preservation, recovery, and post-rebuild usability.
- [ ] Reduce onboarding and sandbox-lifecycle assertions to operator outcomes.
- [ ] Reduce inference and GPU assertions to provider selection, routing, and agent outcomes.
- [ ] Reduce messaging assertions to channel configuration and message behavior.
- [ ] Remove implementation and provenance assertions from platform qualification E2Es while retaining their platform outcomes.
- [ ] Audit the remaining live E2E tail and close any unmapped assertion surface.
- [ ] Enforce the final assertion budget and remove temporary migration exceptions.

The assertion census should land first. Domain children may then proceed in parallel when they do not overlap files. Final enforcement lands after the remediation children.

## Pull-request requirements

Every child pull request must include:

- the affected tests and their observable live outcomes;
- direct and transitive before-and-after assertion counts;
- an assertion-disposition table using the four categories above;
- links to every existing lower-level owner relied upon;
- focused lower-level evidence for assertions moved earlier;
- exact-head trusted GitHub Actions evidence for every changed live target;
- confirmation that no E2E test case, target, job, negative behavior, recovery behavior, cleanup behavior, or security outcome was removed under this epic;
- confirmation that assertions were not laundered into helpers, aggregated objects, snapshots, or generated probes.

## Acceptance criteria

- [ ] The same-scope live suite contains 1,092 or fewer direct assertions.
- [ ] The same-scope live suite averages no more than 11.5 direct assertions per collected test.
- [ ] Live tests and companion modules contain 1,338 or fewer assertions before custom/generated-probe adjustments.
- [ ] The final census also reports custom assertion helpers, product-state `throw`/`fail` conditions, and detectable generated-probe conditions.
- [ ] Every removed assertion has exactly one reviewed disposition.
- [ ] Every assertion moved earlier has a focused lower-level owner that detects the same regression.
- [ ] Assertions described as covered elsewhere link to the exact retained coverage.
- [ ] Assertions deleted as unnecessary include a concrete explanation of why they add no distinct quality signal.
- [ ] Unique live behavior, negative paths, recovery, cleanup, and security outcomes remain covered.
- [ ] No E2E test case, target, or job is removed to meet the budget.
- [ ] No assertion reduction is achieved through aggregation or relocation into helpers or subprocesses.
- [ ] The repository check rejects future growth beyond the final baseline unless an intentional reviewed exception is added.
- [ ] A final exact-commit release-qualification run passes without assertion-related waivers.

## Related work

- #10517 is the existing initial high-cost pruning effort and should be linked as a child of this epic rather than duplicated.
- #9159 owns broader E2E reliability, coverage confidence, and contributor guidance.
- #7912 owns E2E wall time, runner cost, and scheduling.

This epic owns only assertion bloat and the quality-preserving relocation or removal of individual live assertions.

Contributor guide

Open the contributing guide

Research direction

Start with the e2e-live project and test/e2e/live/**/*.test.ts, then inspect the transitive assertion census and the high-cost pruning tracked by #10517. Map each removed assertion to the four required dispositions and verify exact-head GitHub Actions evidence for changed live targets. Done means meeting the 1,092 direct-assertion and 11.5-per-test limits without removing tests, jobs, or distinct behavioral, recovery, cleanup, or security coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.