Epic: improve E2E reliability, coverage confidence, and agent guidance

## Planning status

Work in progress. This Epic records the initial direction from the August 13 developer standup and the August 13 release-qualification incident. Refine the capability slices and measurable baselines before treating this as a complete delivery plan.

## Problem

NemoClaw E2E tests provide essential evidence across real process, sandbox, runtime, inference, and service boundaries. The suite also produces avoidable false failures, takes substantial effort to maintain, and does not give maintainers a clear view of coverage gaps or duplication.

Guidance for agents is spread across README files and code. Agents can add a new E2E test, or expand an existing one, without first proving that deterministic tests and the existing E2E matrix do not already cover the behavior.

## Coverage model from the August 13 standup

The desired execution shape is:

1. Build the image.
2. Generate the matrix.
3. Run the matrix.

The matrix should express supported combinations of:

- agent runtime;
- feature or observable outcome; and
- environment or inference endpoint.

A new runtime, driver, environment, or inference endpoint should usually extend a matrix dimension instead of duplicating test logic. E2E job names should state the observable outcome.

This model does not require every theoretical Cartesian-product combination. The matrix should include supported, meaningful combinations and make intentional exclusions visible.

## Goals

### Improve E2E reliability

- Reduce failures caused by test-harness assumptions, timing, mutable inputs, runner state, or incidental output.
- Preserve evidence that distinguishes product regressions from harness and infrastructure failures.
- Make same-commit reruns and first useful failure boundaries easier to compare.
- Prefer immutable artifacts and bounded readiness checks at external boundaries.

### Make transient E2E failures recoverable without making them invisible

Inventory existing retry behavior, then add or tune bounded, failure-specific retries where operations are idempotent and the failure is demonstrably transient. Report first-attempt failures, retry attempts, passes after retry, and exhausted retries separately. Fix root causes we own; when an external dependency remains unreliable, record ownership and upstream evidence and keep the degraded signal visible.

### Reduce candidate-drift exposure

Release qualification is bound to one exact candidate SHA. A merge to `main` does not change an E2E run already in progress, but it can make that run ineligible as evidence for the new `main` tip and require another full qualification cycle.

| Full E2E run | Duration | Commits merged during the run | First merge after dispatch |
|---|---:|---:|---:|
| [31728212334](https://github.com/NVIDIA/NemoClaw/actions/runs/31728212334) | 55 minutes | 1 | 18 minutes |
| [31743399726](https://github.com/NVIDIA/NemoClaw/actions/runs/31743399726) | 97 minutes | 8 | 15 minutes |
| [31767577321](https://github.com/NVIDIA/NemoClaw/actions/runs/31767577321) | 66 minutes | 2 | 24 minutes |

All three candidates advanced before qualification finished. Reduce the full-suite critical-path runtime so fewer commits merge during the qualification window and less completed evidence is stale on arrival.

- Track total qualification duration, time to the first subsequent merge, commits merged before completion, and rerun count.
- Measure green-evidence latency from candidate selection through passing qualification for that exact candidate.
- Optimize the longest critical-path targets without removing distinct release evidence.
- Treat runtime reduction as a complement to a release stabilization window. Yesterday's 15-to-24-minute first-merge intervals show that a modest speedup alone would not eliminate candidate drift.

### Improve confidence in E2E coverage

- Inventory and visualize the matrix as agent runtime × feature/outcome × environment or inference endpoint.
- Identify missing supported combinations, redundant coverage, and long serial targets.
- Reuse behavior-oriented E2E scenarios across matrix dimensions.
- Keep corner cases, bug regressions, and pure logic in deterministic unit or integration tests unless a real boundary owns the behavior.
- Measure whether additional executions add distinct evidence instead of counting jobs alone.

### Provide better guidance to agents

- Define when a change needs unit, integration, E2E, or multiple levels of evidence.
- Require an agent to check existing E2E ownership before adding or expanding a live test.
- Prefer matrix expansion over duplicated E2E implementations.
- Require outcome-oriented titles and stable behavioral assertions.
- Warn agents against adding dependencies for test convenience without a demonstrated requirement and maintainer decision.
- Put canonical guidance in the contributor workflow and repository agent instructions, not only in implementation comments.

## Initial capability areas

- Remove incidental terminal-copy assertions from shared E2E helpers.
- Classify and measure false failures on repeated runs of the same commit.
- Measure and reduce green-evidence latency and candidate-drift exposure.
- Make the supported E2E matrix inspectable by maintainers and contributors.
- Find serial scenarios that can reuse fixtures or run as independent matrix entries.
- Document the repository's E2E testing philosophy for coding agents.

## Non-goals

- Add a large set of new live E2E tests.
- Require unsupported combinations to run.
- Replace focused deterministic regression tests with live E2E.
- Add test dependencies by default.
- Duplicate the August 13 incident remediation tracked by #9043.
- Duplicate long-running runner reliability and observability work tracked by #7140.

## Completion signals

- The repository defines and tests a stable classification for product, harness, infrastructure, and external-service failures.
- Maintainers can inspect the supported matrix dimensions and intentional exclusions.
- Repeated same-commit qualification provides a measurable false-failure baseline and shows improvement.
- Full qualification reports duration, time to the first subsequent merge, commits merged before completion, and green-evidence latency.
- The measured candidate-drift exposure and green-evidence latency improve from the August 13 baseline.
- Agent guidance explains when and how to add, expand, or avoid E2E tests.
- Each child issue delivers an independently useful improvement with focused deterministic evidence.
- A full release-qualification run demonstrates the combined changes without relying on waived incidental failures.

## Related work

- #9043 tracks the bounded August 13 qualification incident.
- #7140 tracks reliability and observability for long-running runners.