microsoft / microsoft/aspire

Unify E2E test helpers: centralized error handling, logging, and resource assertions

Open
#16,032 1 comment 0 reactions 0 assignees View on GitHub
area-polyglot area-testing
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

## Summary

The CLI E2E tests have grown organically and would benefit from a unified helper layer that centralizes common patterns, error handling, and diagnostics.

## Current state

- `CliE2EAutomatorHelpers.cs` has `AspireStartAsync`/`AspireStopAsync` with good error handling (log dumping, dashboard validation)
- Individual tests duplicate patterns for `aspire init`, `aspire add`, `aspire restore`, `aspire run` + Ctrl+C
- Some tests use `CreateDockerTestTerminal`, others use `CreateTestTerminal` — different environments with different capabilities
- Error diagnostics (log capture, JSON output inspection) are inline in `AspireStartAsync` but not available for other commands
- Resource graph assertions are ad-hoc (e.g., checking file existence, grepping output)

## Proposed improvements

### 1. Reusable resource graph assertions
- Parse `aspire describe --format json` output into typed objects
- Link the shared `ResourceJson` model (`src/Shared/Model/Serialization/ResourceJson.cs`) into the E2E test project
- Build assertion helpers: `AssertResourcesExistAsync`, `AssertResourceHasRelationshipAsync`, `AssertResourceStateAsync`, etc.
- Any E2E test that cares about resource materialization can use these

### 2. Centralized error handling and diagnostics
- Extract the log-dumping logic from `AspireStartAsync` into a reusable helper that any command can use on failure
- Standardize how tests capture and surface CLI logs, detach logs, and JSON output on failure
- Make it easy to add diagnostic capture to any `aspire` command, not just `start`

### 3. Consistent terminal setup patterns
- Document when to use `CreateDockerTestTerminal` vs `CreateTestTerminal`
- Consider a unified factory that picks the right terminal based on what the test needs (Docker socket, Node.js, etc.)

### 4. Verbose logging control
- Add a helper to enable/disable verbose CLI logging (`--log-level`) consistently across tests
- Centralize log level configuration so debugging a failing E2E test is as simple as flipping a flag

## Context

This came up during PR #15901 where we needed to assert un-awaited promise chains actually created resources at runtime. We added `AssertResourcesExistAsync` as a first step, but it uses `node` to parse JSON in the container rather than the typed `ResourceJson` model. A proper solution would make the shared model available to E2E tests.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.