microsoft / microsoft/aspire

Bring non-`dotnet test` runners to parity with `dotnet test` in CI tooling

Open
#16,968 0 comments 0 reactions 1 assignee Claimed by @radical View on GitHub
area-engineering-systems
Dominant language
C#
Stars
6.3k
Forks
991
Avg merge
2d 15h
Merged PRs (30d)
196

Description

Tests that don't run via `dotnet test` are invisible to our CI tooling — no TRX, so they don't land in `All-TestResults`, don't render in the GH step summary (`tools/GenerateTestSummary`), and don't participate in the auto-rerun analyzer's flake-pattern retries (`analyzeTrxFiles` in `.github/workflows/auto-rerun-transient-ci-failures.js`). They also can't use `[QuarantinedTest]` / `[ActiveIssue]`.

#16701 fixes this for polyglot Python/Go/Java/Rust/TypeScript by converting them to xUnit + Docker E2E tests. After it lands, the remaining non-`dotnet test` runners are:

- `typescript-sdk-tests.yml` — vitest unit tests for the TS AppHost SDK
- `extension_tests_win` in `tests.yml` — mocha + `@vscode/test-electron` for the VS Code extension
- `cli_starter_validation_windows` — PowerShell scenario script
- CLI E2E Docker smoke and `eng/scripts/verify-*` scripts

Two paths, not mutually exclusive:

1. **Convert to xUnit** where feasible (the #16701 model). Not applicable to vitest/mocha — they test JS/TS source directly.
2. **Teach `tools/Aspire.TestTools` (`TrxReader`, `TestSummaryGenerator`) and the auto-rerun analyzer (`extractFailedTestsFromTrx`) to also read JUnit XML.** vitest and mocha emit JUnit natively (`--reporter=junit`, `mocha-junit-reporter`). Pester covers the PS cases if we want them surfaced as tests.

Goal: every test runner produces results that flow into the same downstream surfaces (`All-TestResults`, step summary, auto-rerun, quarantine/active-issue semantics) — parity with `dotnet test`.

Related: #16701.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.