Unify E2E test helpers: centralized error handling, logging, and resource assertions
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
## Summary
The CLI E2E tests have grown organically and would benefit from a unified helper layer that centralizes common patterns, error handling, and diagnostics.
## Current state
- `CliE2EAutomatorHelpers.cs` has `AspireStartAsync`/`AspireStopAsync` with good error handling (log dumping, dashboard validation)
- Individual tests duplicate patterns for `aspire init`, `aspire add`, `aspire restore`, `aspire run` + Ctrl+C
- Some tests use `CreateDockerTestTerminal`, others use `CreateTestTerminal` — different environments with different capabilities
- Error diagnostics (log capture, JSON output inspection) are inline in `AspireStartAsync` but not available for other commands
- Resource graph assertions are ad-hoc (e.g., checking file existence, grepping output)
## Proposed improvements
### 1. Reusable resource graph assertions
- Parse `aspire describe --format json` output into typed objects
- Link the shared `ResourceJson` model (`src/Shared/Model/Serialization/ResourceJson.cs`) into the E2E test project
- Build assertion helpers: `AssertResourcesExistAsync`, `AssertResourceHasRelationshipAsync`, `AssertResourceStateAsync`, etc.
- Any E2E test that cares about resource materialization can use these
### 2. Centralized error handling and diagnostics
- Extract the log-dumping logic from `AspireStartAsync` into a reusable helper that any command can use on failure
- Standardize how tests capture and surface CLI logs, detach logs, and JSON output on failure
- Make it easy to add diagnostic capture to any `aspire` command, not just `start`
### 3. Consistent terminal setup patterns
- Document when to use `CreateDockerTestTerminal` vs `CreateTestTerminal`
- Consider a unified factory that picks the right terminal based on what the test needs (Docker socket, Node.js, etc.)
### 4. Verbose logging control
- Add a helper to enable/disable verbose CLI logging (`--log-level`) consistently across tests
- Centralize log level configuration so debugging a failing E2E test is as simple as flipping a flag
## Context
This came up during PR #15901 where we needed to assert un-awaited promise chains actually created resources at runtime. We added `AssertResourcesExistAsync` as a first step, but it uses `node` to parse JSON in the container rather than the typed `ResourceJson` model. A proper solution would make the shared model available to E2E tests.
Contributor guide
Assessment
This issue has not been assessed yet.