Track full Native AOT support for dotnet test
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 1.3k
- PR merge metrics
- PR metrics pending
Description
### Is your feature request related to a problem?
Follow-up to [#55971](https://github.com/dotnet/sdk/pull/55971), which adds a Native AOT fast path for build-free Microsoft.Testing.Platform (MTP) invocations with an explicit `--test-modules` expression.
The rest of the `dotnet test` verb still enters the managed CLI. The current boundary is intentional and safe, but it leaves project, solution, directory, file-based, VSTest, help, and device paths without a Native AOT implementation.
| Runner / input shape | Current behavior after #55971 |
| --- | --- |
| MTP, explicit non-empty `--test-modules` with the supported option set | Native AOT |
| MTP, positional `.dll` / `.exe` module shorthand | Managed fallback |
| MTP, project / solution / solution filter / solution XML / directory discovery | Managed fallback |
| MTP, file-based C# input | Managed fallback |
| MTP project inputs with build, restore, `--no-build`, or `--no-restore` | Managed fallback; even `--no-build` still evaluates projects and computes run properties |
| MTP `--device` / `--list-devices` | Managed fallback |
| VSTest, direct built `.dll` / `.exe` | Managed fallback |
| VSTest, project / solution / directory | Managed fallback |
| `dotnet test --help` | Managed fallback |
The current eligibility check is deliberately conservative: [`AotTestCommand`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Commands/Test/AotTestCommand.cs#L27-L95) validates the runner, explicit module expression, root options, command options, and unmatched tokens before printing the tier marker or starting MTP orchestration. The AOT parser otherwise assigns managed fallback recursively and overrides only known-safe command shapes ([`Parser.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Parser.cs#L205-L233)).
### Describe the solution you'd like
Make progressively more `dotnet test` shapes execute through the Native AOT bridge while preserving exact parsing, output, exit codes, cancellation, telemetry, artifacts, first-run behavior, and managed fallback semantics.
#### Eligibility and commit-point constraints
A shape may enter the native tier only when support is known **before any observable side effect**. [`NativeEntryPoint`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet-aot/NativeEntryPoint.cs#L181-L263) can safely catch `CommandNotAvailableInAotException` and host `dotnet.dll` only before the native command has built, restored, launched, deployed, created IPC state, or emitted user-visible output. A native path cannot run an external build, discover later that evaluation is unsupported, and then replay the command through managed fallback.
First-run setup is also a global prerequisite: if the AOT first-run implementation cannot complete, `NativeEntryPoint` deliberately defers the whole invocation before command execution ([lines 190-208](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet-aot/NativeEntryPoint.cs#L190-L208)).
#### Hard blockers
- [ ] **Replace arbitrary in-process MSBuild use on MTP project paths.** `MSBuildHandler` is entirely marked `RequiresDynamicCode` and dispatches project/solution handling through `MSBuildUtility` ([`MSBuildHandler.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Commands/Test/MTP/MSBuildHandler.cs#L11-L41)). Project, solution, and file-based entry points are also marked `RequiresDynamicCode` and use MSBuild evaluation/build APIs ([`MSBuildUtility.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Commands/Test/MTP/MSBuildUtility.cs#L34-L171)).
- [ ] **Preserve arbitrary SDK/task extensibility.** Native AOT disables dynamic SDK-resolver loading and unrestricted property functions; only the workload and NuGet resolvers are registered statically ([`MSBuildSdkResolverRegistration.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet-aot/MSBuildSdkResolverRegistration.cs#L10-L40), [`MSBuildEvaluationTests.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/test/dotnet-aot.Tests/MSBuildEvaluationTests.cs#L27-L41)). The existing AOT test proves only evaluation of a minimal stock `Microsoft.NET.Sdk` project with directory imports disabled ([lines 63-124](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/test/dotnet-aot.Tests/MSBuildEvaluationTests.cs#L63-L124)); it does not prove custom SDK resolvers, custom tasks, or target execution.
- [ ] **Move run-property and device discovery across a stable process boundary.** The managed path may run `ComputeAvailableDevices`, `DeployToDevice`, and `ComputeRunArguments`, then reads mutated project state into `RunProperties` ([`SolutionAndProjectUtility.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Commands/Test/MTP/SolutionAndProjectUtility.cs#L455-L586), [lines 589-713](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Commands/Test/MTP/SolutionAndProjectUtility.cs#L589-L713)). Those targets can load arbitrary managed task assemblies and cannot be assumed safe inside a native image.
- [ ] **Define preflight/fallback behavior for failures.** The design must guarantee that every unsupported shape is rejected before build/restore/deploy/test launch, and that errors after commitment are returned directly rather than replayed through managed fallback.
The likely remediation is an out-of-process MSBuild target that emits a versioned, machine-readable test-run manifest. The manifest should contain the selected project/TFM/module graph, command, arguments, working directory, runtime environment, launch settings, target path, artifact metadata, and device mapping. Native orchestration can then consume that manifest with source-generated serialization. The managed implementation should consume the same contract first so parity is tested before switching eligibility.
#### Independently feasible work
- [ ] **Add direct VSTest `.dll` / `.exe` support.** A controlled experiment linked `VSTestArgumentConverter` and the existing `VSTestForwardingApp`, committed only after finding a built source, and preserved artifact collection/post-processing. It built and Native AOT-published with 0 warnings; in a complete separated SDK layout, `dn` listed two tests, ran one passing test (`1/1`, exit `0`), propagated one failing test (`1/1`, exit `1`), and self-located the SDK root successfully. Debug native size increased from 35,077,120 to 35,183,616 bytes: **+106,496 bytes (+0.304%)**. This experiment was reverted after measurement.
- [ ] **Add VSTest project/solution forwarding.** This path already shells out to MSBuild and VSTest rather than evaluating arbitrary projects in the AOT process ([`TestCommand.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Commands/Test/VSTest/TestCommand.cs#L44-L56)); close and validate its `RestoringCommand`, terminal-logger, environment, and artifact-processing dependencies independently.
- [ ] **Support MTP positional module shorthand.** Reproduce managed precedence and ambiguity checks exactly before treating an existing positional `.dll` / `.exe` as the already-supported module tier. The AOT branch currently omits positional inference and all MSBuild arguments by construction ([`TestCommandOptions.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Commands/Test/MTP/TestCommandOptions.cs#L21-L76), [lines 99-125](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Commands/Test/MTP/TestCommandOptions.cs#L99-L125)).
- [ ] **Split VSTest help from MTP dynamic help.** The AOT help action currently falls back for the entire `test` subtree because MTP builds/runs test applications to collect extension help ([`Parser.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Parser.cs#L271-L300)). Static VSTest help may be eligible independently; MTP help depends on the project-manifest/build design.
- [ ] **Define a validated `--no-build` manifest/cache tier.** A cache must prove identity and freshness before commitment and include every run property that MSBuild targets can affect. A missing/stale/incomplete cache must fall back before any native side effect.
#### Project, solution, directory, file, and device checklist
- [ ] Specify and version the out-of-process manifest schema and producer target.
- [ ] Make managed MTP consume the manifest and establish parity baselines.
- [ ] Support explicit project input with `--no-build`, then build and restore.
- [ ] Support `.sln`, `.slnf`, and `.slnx`, including configuration/platform inclusion semantics.
- [ ] Support directory discovery and preserve current ambiguity/error messages ([`ValidationUtility.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Commands/Test/MTP/ValidationUtility.cs#L57-L155)).
- [ ] Support file-based C# input, including its virtual-project build/cache/source-editing closure.
- [ ] Support multi-targeting, parallel/sequential grouping, launch settings, environment variables, and `DOTNET_ROOT_` selection.
- [ ] Support `--list-devices`, `--device`, interactive TFM/device selection, per-TFM device builds, deployment, and device-provided RIDs ([`MicrosoftTestingPlatformTestCommand.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/Commands/Test/MTP/MicrosoftTestingPlatformTestCommand.cs#L744-L900)).
- [ ] Preserve binary logging and MSBuild error/warning output. `MSBuildSession` currently creates in-process loggers and assumes managed, untrimmed MSBuild behavior ([`MSBuildSession.cs`](https://github.com/dotnet/sdk/blob/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d/src/Cli/dotnet/MSBuildSession.cs#L252-L281)).
#### Cross-cutting acceptance criteria
- [ ] Clean source closure build and Native AOT publish have no new trim/AOT warnings and no warning suppressions justified only by managed/untrimmed assumptions.
- [ ] Every tier has an explicit, side-effect-free eligibility predicate and a named commit-point marker.
- [ ] Native and managed runs match exit code, meaningful stdout/stderr, output files, environment, test count, and build/restore/deploy side effects.
- [ ] Passing, failing, skipped, zero-test, timeout, maximum-failure, minimum-expected-test, cancellation, malformed response-file, and child-process-crash cases are covered.
- [ ] Results layouts, TRX/log attachments, test maps/affected tests, and artifact post-processing are covered with actual artifacts rather than compilation-only assertions.
- [ ] Telemetry validates runner/tier/command tags, success/failure, duration, cancellation, and exactly-once emission; fallback must not duplicate native and managed telemetry.
- [ ] First-run-required invocations still fall back before commitment and execute exactly once.
- [ ] `DOTNET_CLI_ENABLEAOT=false` and unsupported options preserve managed behavior.
- [ ] Flat development layout, real redist SDK layout, separated versioned SDK layout, host-provided `sdk_dir`, and module self-location are covered.
- [ ] Native size deltas are measured in Release and reviewed per tier.
- [ ] The platform matrix covers at least Windows x64/Arm64, Linux x64/Arm64, and macOS Arm64 (plus macOS x64 while supported).
### Alternatives you've considered
1. **Compile the current in-process MSBuild path into Native AOT.** Rejected as the default design. A source-closure experiment that added the four excluded MTP/MSBuild files plus `MSBuildSession` first failed with 14 missing AOT-excluded APIs/types. Expanding that closure exposed additional file-based compiler/editor/restore dependencies. More importantly, successful compilation would not make arbitrary resolver/task loading or target execution safe.
2. **Run build out of process, then evaluate in process.** Unsafe: the build is already an observable commit point, so an evaluation failure cannot transparently replay through managed fallback.
3. **Keep all project paths on managed fallback indefinitely.** Correct and safe, but does not achieve full-verb compatibility. It remains the required behavior until a complete tier is proven.
4. **Start with cache-only `--no-build`.** Potentially viable if the cache is a complete, versioned run contract with conservative freshness validation; otherwise it can silently diverge from `ComputeRunArguments`, launch settings, device targets, and custom project logic.
### Additional context
Validation performed on Windows x64 at commit [`7c8c002d`](https://github.com/dotnet/sdk/commit/7c8c002dced2b8ac7528f86d37ac77cd4d736b6d):
| Validation | Result |
| --- | --- |
| Full repository `build.cmd` | Passed, 0 warnings / 0 errors |
| Clean `dotnet-aot` and `dotnet-aot.Tests` closure builds | Passed, 0 warnings / 0 errors |
| Focused managed `AotParserTests` | 58 executed, 58 passed, 0 failed, 0 skipped |
| Product Native AOT publish (`win-x64`, Debug) | Passed, 0 warnings; 35,077,120-byte library |
| Real `dn`, built redist SDK, MTP explicit module | 1 executed, 1 passed, exit 0 |
| Real `dn`, built redist SDK, unsupported MTP project input | Managed fallback; 1 executed, 1 passed, exit 0 |
| Separated SDK root with self-location, MTP explicit module | 1 executed, 1 passed, exit 0 |
| Direct VSTest experiment | List: 2 tests; pass: 1/1 exit 0; intentional failure: 1/1 exit 1; 0 build/publish warnings; +106,496 bytes Debug |
| Native-published `dotnet-aot.Tests` host | Process crashed after reporting 9 failures; no TRX/final counts were produced |
The native-test-host failures are presently a **harness/layout gap**, not a demonstrated `dotnet test` product blocker. `run-aot-tests.ps1` loads `dotnet-aot` from the test artifact directory while passing the bootstrap SDK as `sdk_dir`, violating the debug self-location assertion that the loaded library and SDK directory identify the same versioned SDK. The stock Windows `run-dn.ps1 -Mode Compare` synthetic layout also does not put `hostfxr` on the native loader search path, so its managed half fails even though manual execution from the complete built redist layout succeeds. Fix the harnesses before using those failures as acceptance evidence.
Only Windows x64 was exercised. Windows Arm64, Linux, and macOS remain validation gaps. Cancellation, telemetry payloads, device paths, dynamic MTP help, and real artifact post-processing were mapped in source but were not proven end to end in this investigation.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with AotTestCommand.cs, Parser.cs, and NativeEntryPoint.cs to understand current eligibility checks, fallback behavior, and the commit point. Then read the independently feasible VSTest and MTP paths named in the issue, along with their referenced tests and entry points. Done requires a narrowly scoped tier with side-effect-free eligibility, managed parity, Native AOT publication without new warnings, and the listed behavioral tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- build-system, cli, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100