Bring non-`dotnet test` runners to parity with `dotnet test` in CI tooling
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
Tests that don't run via `dotnet test` are invisible to our CI tooling — no TRX, so they don't land in `All-TestResults`, don't render in the GH step summary (`tools/GenerateTestSummary`), and don't participate in the auto-rerun analyzer's flake-pattern retries (`analyzeTrxFiles` in `.github/workflows/auto-rerun-transient-ci-failures.js`). They also can't use `[QuarantinedTest]` / `[ActiveIssue]`.
#16701 fixes this for polyglot Python/Go/Java/Rust/TypeScript by converting them to xUnit + Docker E2E tests. After it lands, the remaining non-`dotnet test` runners are:
- `typescript-sdk-tests.yml` — vitest unit tests for the TS AppHost SDK
- `extension_tests_win` in `tests.yml` — mocha + `@vscode/test-electron` for the VS Code extension
- `cli_starter_validation_windows` — PowerShell scenario script
- CLI E2E Docker smoke and `eng/scripts/verify-*` scripts
Two paths, not mutually exclusive:
1. **Convert to xUnit** where feasible (the #16701 model). Not applicable to vitest/mocha — they test JS/TS source directly.
2. **Teach `tools/Aspire.TestTools` (`TrxReader`, `TestSummaryGenerator`) and the auto-rerun analyzer (`extractFailedTestsFromTrx`) to also read JUnit XML.** vitest and mocha emit JUnit natively (`--reporter=junit`, `mocha-junit-reporter`). Pester covers the PS cases if we want them surfaced as tests.
Goal: every test runner produces results that flow into the same downstream surfaces (`All-TestResults`, step summary, auto-rerun, quarantine/active-issue semantics) — parity with `dotnet test`.
Related: #16701.
Contributor guide
Assessment
This issue has not been assessed yet.